ntfy/docs/troubleshooting.md

Troubleshooting

This page lists a few suggestions of what to do when things don't work as expected. This is not a complete list. If this page does not help, feel free to drop by the Discord or Matrix and ask there. We're happy to help.

ntfy server

If you host your own ntfy server, and you're having issues with any component, it is always helpful to enable debugging/tracing in the server. You can find detailed instructions in the Logging & Debugging section, but it ultimately boils down to setting log-level: debug or log-level: trace in the server.yml file:

=== "server.yml (debug)" yaml log-level: debug

=== "server.yml (trace)" yaml log-level: trace

If you're using environment variables, set NTFY_LOG_LEVEL=debug (or trace) instead. You can also pass --debug or --trace to the ntfy serve command, e.g. ntfy serve --trace. If you're using systemd (i.e. systemctl) to run ntfy, you can look at the logs using journalctl -u ntfy -f. The logs will look something like this:

=== "Example logs (debug)" $ ntfy serve --debug 2023/03/20 14:45:38 INFO Listening on :2586[http] :1025[smtp], ntfy 2.1.2, log level is DEBUG (tag=startup) 2023/03/20 14:45:38 DEBUG Waiting until 2023-03-21 00:00:00 +0000 UTC to reset visitor stats (tag=resetter) 2023/03/20 14:45:39 DEBUG Rate limiters reset for visitor (visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=0, visitor_messages_limit=500, visitor_messages_remaining=500, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2023-03-20T14:45:39.7-04:00) 2023/03/20 14:45:39 DEBUG HTTP request started (http_method=POST, http_path=/mytopic, tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=0, visitor_messages_limit=500, visitor_messages_remaining=500, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2023-03-20T14:45:39.7-04:00) 2023/03/20 14:45:39 DEBUG Received message (http_method=POST, http_path=/mytopic, message_body_size=2, message_delayed=false, message_email=, message_event=message, message_firebase=true, message_id=EZu6i2WZjH0v, message_sender=127.0.0.1, message_time=1679337939, message_unifiedpush=false, tag=publish, topic=mytopic, topic_last_access=2023-03-20T14:45:38.319-04:00, topic_subscribers=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0002132248, visitor_seen=2023-03-20T14:45:39.7-04:00) 2023/03/20 14:45:39 DEBUG Adding message to cache (http_method=POST, http_path=/mytopic, message_body_size=2, message_event=message, message_id=EZu6i2WZjH0v, message_sender=127.0.0.1, message_time=1679337939, tag=publish, topic=mytopic, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000259165, visitor_seen=2023-03-20T14:45:39.7-04:00) 2023/03/20 14:45:39 DEBUG HTTP request finished (http_method=POST, http_path=/mytopic, tag=http, time_taken_ms=2, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0004147334, visitor_seen=2023-03-20T14:45:39.7-04:00) 2023/03/20 14:45:39 DEBUG Wrote 1 message(s) in 8.285712ms (tag=message_cache) ...

=== "Example logs (trace)" ``` $ ntfy serve --trace 2023/03/20 14:40:42 INFO Listening on :2586[http] :1025[smtp], ntfy 2.1.2, log level is TRACE (tag=startup) 2023/03/20 14:40:42 DEBUG Waiting until 2023-03-21 00:00:00 +0000 UTC to reset visitor stats (tag=resetter) 2023/03/20 14:40:59 DEBUG Rate limiters reset for visitor (visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=0, visitor_messages_limit=500, visitor_messages_remaining=500, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2023-03-20T14:40:59.893-04:00) 2023/03/20 14:40:59 TRACE HTTP request started (http_method=POST, http_path=/mytopic, http_request=POST /mytopic HTTP/1.1 User-Agent: curl/7.81.0 Accept: / Content-Length: 2 Content-Type: application/x-www-form-urlencoded

hi, tag=http, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=0, visitor_messages_limit=500, visitor_messages_remaining=500, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=60, visitor_seen=2023-03-20T14:40:59.893-04:00)
2023/03/20 14:40:59 TRACE Received message (http_method=POST, http_path=/mytopic, message_body={
  "id": "Khaup1RVclU3",
  "time": 1679337659,
  "expires": 1679380859,
  "event": "message",
  "topic": "mytopic",
  "message": "hi"
}, message_body_size=2, message_delayed=false, message_email=, message_event=message, message_firebase=true, message_id=Khaup1RVclU3, message_sender=127.0.0.1, message_time=1679337659, message_unifiedpush=false, tag=publish, topic=mytopic, topic_last_access=2023-03-20T14:40:59.893-04:00, topic_subscribers=0, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0001785048, visitor_seen=2023-03-20T14:40:59.893-04:00)
2023/03/20 14:40:59 DEBUG Adding message to cache (http_method=POST, http_path=/mytopic, message_body_size=2, message_event=message, message_id=Khaup1RVclU3, message_sender=127.0.0.1, message_time=1679337659, tag=publish, topic=mytopic, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0002044368, visitor_seen=2023-03-20T14:40:59.893-04:00)
2023/03/20 14:40:59 DEBUG HTTP request finished (http_method=POST, http_path=/mytopic, tag=http, time_taken_ms=1, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.000220502, visitor_seen=2023-03-20T14:40:59.893-04:00)
2023/03/20 14:40:59 TRACE No stream or WebSocket subscribers, not forwarding (message_body_size=2, message_event=message, message_id=Khaup1RVclU3, message_sender=127.0.0.1, message_time=1679337659, tag=publish, topic=mytopic, visitor_auth_limiter_limit=0.016666666666666666, visitor_auth_limiter_tokens=10, visitor_emails=0, visitor_emails_limit=12, visitor_emails_remaining=12, visitor_id=ip:127.0.0.1, visitor_ip=127.0.0.1, visitor_messages=1, visitor_messages_limit=500, visitor_messages_remaining=499, visitor_request_limiter_limit=0.2, visitor_request_limiter_tokens=59.0002369212, visitor_seen=2023-03-20T14:40:59.893-04:00)
2023/03/20 14:41:00 DEBUG Wrote 1 message(s) in 9.529196ms (tag=message_cache)
...
```

Android app

On Android, you can turn on logging in the settings under Settings → Record logs. This will store up to 1,000 log entries, which you can then copy or upload.

{ width=400 }

Recording logs on Android

When you copy or upload the logs, you can censor them to make it easier to share them with others. ntfy will replace all topics and hostnames with fruits. Here's an example:

This is a log of the ntfy Android app. The log shows up to 1,000 entries.
Server URLs (aside from ntfy.sh) and topics have been replaced with fruits 🍌🥝🍋🥥🥑🍊🍎🍑.

Device info:
--
ntfy: 1.16.0 (play)
OS: 4.19.157-perf+
Android: 13 (SDK 33)
...

Logs
--

1679339199507 2023-03-20 15:06:39.507 D NtfyMainActivity Battery: ignoring optimizations = true (we want this to be true); instant subscriptions = true; remind time reached = true; banner = false
1679339199507 2023-03-20 15:06:39.507 D NtfySubscriberMgr Enqueuing work to refresh subscriber service
1679339199589 2023-03-20 15:06:39.589 D NtfySubscriberMgr ServiceStartWorker: Starting foreground service with action START (work ID: a7eeeae9-9356-40df-afbd-236e5ed10a0b)
1679339199602 2023-03-20 15:06:39.602 D NtfySubscriberService onStartCommand executed with startId: 262
1679339199602 2023-03-20 15:06:39.602 D NtfySubscriberService using an intent with action START
1679339199629 2023-03-20 15:06:39.629 D NtfySubscriberService Refreshing subscriptions
1679339199629 2023-03-20 15:06:39.629 D NtfySubscriberService - Desired connections: [ConnectionId(baseUrl=https://ntfy.sh, topicsToSubscriptionIds={avocado=23801492, lemon=49013182, banana=1309176509201171073, peach=573300885184666424, pineapple=-5956897229801209316, durian=81453333, starfruit=30489279, fruit12=82532869}), ConnectionId(baseUrl=https://orange.example.com, topicsToSubscriptionIds={apple=4971265, dragonfruit=66809328})]
1679339199629 2023-03-20 15:06:39.629 D NtfySubscriberService - Active connections: [ConnectionId(baseUrl=https://orange.example.com, topicsToSubscriptionIds={apple=4971265, dragonfruit=66809328}), ConnectionId(baseUrl=https://ntfy.sh, topicsToSubscriptionIds={avocado=23801492, lemon=49013182, banana=1309176509201171073, peach=573300885184666424, pineapple=-5956897229801209316, durian=81453333, starfruit=30489279, fruit12=82532869})]
...

To get live logs, or to get more advanced access to an Android phone, you can use adb. After you install and enable adb debugging, you can get detailed logs like so:

# Connect to phone (enable Wireless debugging first)
adb connect 192.168.1.137:39539

# Print all logs; you may have to pass the -s option
adb logcat
adb -s 192.168.1.137:39539 logcat

# Only list ntfy logs
adb logcat --pid=$(adb shell pidof -s io.heckel.ntfy)
adb -s 192.168.1.137:39539 logcat --pid=$(adb -s 192.168.1.137:39539 shell pidof -s io.heckel.ntfy)

Web app

The web app logs everything to the developer console, which you can open by pressing the F12 key on your keyboard.


Web app logs in the developer console

iOS app

Sorry, there is no way to debug or get the logs from the iOS app (yet), outside of running the app in Xcode.