A complete investigation workflow
Start with one reproducible symptom and an exact UTC window. Use Call-ID and CSeq to bind evidence from Asterisk, sngrep and packet capture to the same dialog and transaction. Identify the first hop where observed behavior diverges from expected behavior. Do not begin by enabling every debug facility or assigning blame based on a response phrase.
Define symptom and UTC test window
date -u +%Y-%m-%dT%H:%M:%SZ
sudo asterisk -rx "core show uptime"
sudo asterisk -rx "pjsip show endpoints"Write the symptom directionally: “outbound INVITE receives 403 from 203.0.113.10” or “Asterisk sends RTP but receives none.” Record the authorized test origin and destination in a protected incident record, then sanitize them for external sharing. Time synchronization matters because packet and application logs otherwise cannot be aligned reliably.
Use Asterisk logging for one test
asterisk -rvvv
core set verbose 4
core set debug 0
pjsip set logger on
# place exactly one controlled test call
pjsip set logger off
core set verbose 3Increase verbosity only enough to expose dialplan and channel decisions. PJSIP logging shows complete SIP messages and may expose URIs, PAI, digest fields, SDP, Identity tokens and topology. Turn it off immediately after the test. Search the captured terminal or protected log for the Call-ID, then follow CSeq methods in order.
Call-ID: debug-001@example.com
CSeq: 101 INVITE
<--- Transmitting SIP request to UDP:203.0.113.10:5060 --->
<--- Received SIP response (403 Forbidden) from UDP:203.0.113.10:5060 --->Read the SIP ladder with sngrep
sudo sngrep -d any host 203.0.113.10
# In the UI, filter by the exact Call-ID or sanitized endpoint pair.Asterisk Peer 203.0.113.10
| INVITE CSeq:101 |
|------------------------>|
| 100 Trying |
|<------------------------|
| 403 Forbidden |
|<------------------------|
| ACK |
|------------------------>|The ladder makes hop direction visible, but an on-box capture cannot show what happened inside a downstream network. The packet source of the 403 identifies the immediate responder, which may be a proxy rather than the ultimate policy engine. Inspect Via and Record-Route carefully before inferring topology.
Capture only the relevant packets
sudo tcpdump -ni any host 203.0.113.10 and port 5060 -s 0 -w /tmp/sip-debug-001.pcap
sudo tcpdump -ni any host 203.0.113.10 and udp portrange 10000-20000 -s 96 -w /tmp/rtp-debug-001.pcap
sudo tcpdump -nnr /tmp/sip-debug-001.pcapUse -s 0 only when full signaling bodies are necessary. A short snap length can confirm RTP direction without retaining full payload, but validate that it still meets the diagnostic goal. Stop the capture promptly. TLS hides SIP content from packet capture; correlate TCP/TLS sessions with Asterisk’s authorized logs instead of weakening encryption.
Interpret a normal authentication challenge
REGISTER sip:sip.example.net SIP/2.0
Via: SIP/2.0/UDP 198.51.100.20:5060;branch=z9hG4bK-reg1
Call-ID: reg-001@example.com
CSeq: 1 REGISTER
SIP/2.0 401 Unauthorized
Call-ID: reg-001@example.com
CSeq: 1 REGISTER
WWW-Authenticate: Digest realm="example",nonce="REDACTED"
REGISTER sip:sip.example.net SIP/2.0
Call-ID: reg-001@example.com
CSeq: 2 REGISTER
Authorization: Digest username="REDACTED",response="REDACTED"A first 401 can be expected: it supplies the challenge. The meaningful question is whether the next REGISTER carries appropriate authorization and receives success. Repeated challenges or a final rejection require realm, username, clock, nonce handling and account-policy evidence. Never publish the Authorization header.
Attribute an error to the observed hop
SIP/2.0 503 Service Unavailable
Via: SIP/2.0/UDP 198.51.100.20:5060;branch=z9hG4bK-call1
Call-ID: debug-002@example.com
CSeq: 101 INVITE
Retry-After: 30
Reason: Q.850;cause=41;text="Temporary failure"The packet source and Via transaction identify where this response entered the capture. Retry-After conveys a suggested interval when present; it is not permission for an uncontrolled retry storm. The Reason header adds context but does not override direct resource, routing and account evidence.
Correlate sources without leaking secrets
sudo journalctl -u asterisk --since '2026-09-13 16:00:00 UTC' --until '2026-09-13 16:02:00 UTC' --no-pager | grep -F 'debug-002@example.com'
sudo grep -F 'debug-002@example.com' /var/log/asterisk/fullUse fixed-string matching for a known Call-ID. If logs omit SIP headers, correlate unique channel identifiers and timestamps cautiously. Redact full Identity headers, digest material, Contact user parts, customer numbers and private addresses. Limit incident access and retention; debugging convenience does not eliminate privacy obligations.
Root-cause classification matrix
| Observed first divergence | Class | Next action |
|---|---|---|
| No outbound INVITE leaves Asterisk | Local dialplan/object/routing | Inspect channel, endpoint and dialplan decision. |
| INVITE leaves; no response returns | Network/peer reachability | Inspect retransmissions and captures at boundary. |
| 403 returns from known hop | Admission/identity/account/destination policy | Use 403 trace matrix. |
| 488 returns with SDP context | Offer/answer or encryption | Compare payloads and media profiles. |
| 503 generated locally | Local resource or routing | Inspect channels, endpoints and dialplan. |
| 200/ACK succeeds; RTP missing one way | Media/NAT/firewall | Use packet-driven NAT/RTP guide. |
Close debugging safely
Turn off PJSIP and RTP loggers, return verbosity to the approved baseline, stop captures, verify disk usage and store only the minimum protected evidence. Remove temporary pcap files according to retention requirements after the case record is complete. If any configuration changed during isolation, restore the reviewed baseline and repeat one known-good call.
Related engineering resources
Expected output review and escalation boundary
A healthy investigation produces a single ordered evidence set: synchronized UTC timestamp, one Call-ID, every CSeq transaction, the immediate source of each response, the selected PJSIP endpoint, and media direction when the call answered. It should be possible for a second engineer to replay the reasoning without access to an unrestricted capture. If two tools disagree, compare capture interface, clock, transport encryption and whether a proxy rewrote the dialog identifiers before assuming data loss.
Escalate only the minimum sanitized evidence necessary. For a signaling rejection, include request method, response code, immediate source, selected route and Reason value; for media, include negotiated addresses/ports and directional packet counts. State which observations are direct and which are inferred. This prevents a downstream team from treating an Asterisk channel translation as the original network cause.
Authoritative references
Rollback after evidence collection
Disable verbose PJSIP logging, stop packet capture, remove temporary capture files under the approved retention process, and restore any temporary logger or firewall scope. Re-run one controlled call after rollback to confirm normal operation.
Implementation closure
Directional symptom, UTC window, test endpoints, Call-ID, capture interface, peer addresses, authorization to collect traffic, and secure evidence storage.
Temporary Asterisk logger level and narrowly filtered packet capture; persistent call configuration should change only after evidence identifies a cause.
Enable logging for one test, run scoped sngrep/tcpdump capture, stop collection promptly, then restore logger state without restarting Asterisk.
CLI messages and packets correlate by Call-ID/CSeq and identify the first divergent hop or media boundary.
If correlation is incomplete, narrow the test window and capture point; do not increase logging scope indefinitely or guess at configuration.
Stop when captures include unrelated traffic, credentials cannot be sanitized, storage is not approved, or the next test lacks a falsifiable hypothesis.
The evidence set identifies a responsible boundary, sensitive material is controlled, temporary diagnostics are removed, and the next action has a verification test.