From e18708941a5867bf655457b16f1c12875b33fcc8 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Wed, 16 Jan 2019 09:31:11 +0100 Subject: [PATCH 1/8] Restructure FTLDNS->Configuration page to list keys in sections Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 88 ++++++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 38 deletions(-) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index d1b9f9e..28b1271 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -2,10 +2,31 @@ You can create a file `/etc/pihole/pihole-FTL.conf` that will be read by *FTL*DN Possible settings (**the option shown first is the default**): -### SOCKET_LISTENING -`SOCKET_LISTENING=localonly|all` +On which port should FTL be listening? -Listen only for local socket connections or permit all connections +## DNS settings + +### BLOCKINGMODE +`BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN` + +## Statistics settings + +### MAXLOGAGE +`MAXLOGAGE=24.0` + +Up to how many hours of queries should be imported from the database and logs? Maximum is 744 (31 days) + +### PRIVACYLEVEL +`PRIVACYLEVEL=0|1|2|3|4` + +Which privacy level is used? + +**[More details](privacylevels.md)** + +### IGNORE_LOCALHOST +`IGNORE_LOCALHOST=no|yes` + +Should `FTL` ignore queries coming from the local machine? ### AAAA_QUERY_ANALYSIS `AAAA_QUERY_ANALYSIS=yes|no` @@ -17,6 +38,22 @@ Allow `FTL` to analyze AAAA queries from pihole.log? Should `FTL` only analyze A and AAAA queries? +How should `FTL` reply to blocked queries? + +**[More details](blockingmode.md)** + +## Socket settings + +### SOCKET_LISTENING +`SOCKET_LISTENING=localonly|all` + +Listen only for local socket connections or permit all connections + +### FTLPORT +`FTLPORT=4711` + +## Host name resolution + ### RESOLVE_IPV6 `RESOLVE_IPV6=yes|no` @@ -27,6 +64,15 @@ Should `FTL` try to resolve IPv6 addresses to host names? Should `FTL` try to resolve IPv4 addresses to host names? +## Database settings + +### DBIMPORT +`DBIMPORT=yes|no` + +Should `FTL` load information from the database on startup to be aware of the most recent history? + +**[More details](database.md)** + ### MAXDBDAYS `MAXDBDAYS=365` @@ -49,34 +95,7 @@ Specify path and filename of FTL's SQLite3 long-term database. Setting this to ` **[More details](database.md)** -### MAXLOGAGE -`MAXLOGAGE=24.0` - -Up to how many hours of queries should be imported from the database and logs? Maximum is 744 (31 days) - -### FTLPORT -`FTLPORT=4711` - -On which port should FTL be listening? - -### PRIVACYLEVEL -`PRIVACYLEVEL=0|1|2|3|4` - -Which privacy level is used? - -**[More details](privacylevels.md)** - -### IGNORE_LOCALHOST -`IGNORE_LOCALHOST=no|yes` - -Should `FTL` ignore queries coming from the local machine? - -### BLOCKINGMODE -`BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN` - -How should `FTL` reply to blocked queries? - -**[More details](blockingmode.md)** +## Debugging options ### REGEX_DEBUGMODE ``` @@ -87,12 +106,5 @@ Controls if *FTL*DNS should print extended details about regex matching into `pi **[More details](regex/overview.md)** -### DBIMPORT -`DBIMPORT=yes|no` - -Should `FTL` load information from the database on startup to be aware of the most recent history? - -**[More details](database.md)** - {!abbreviations.md!} From d4fa190243c7c9a3d8d568c54aae4ab4a40dddf7 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Wed, 16 Jan 2019 09:49:15 +0100 Subject: [PATCH 2/8] Add configuration documentation for new debug flags Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 62 ++++++++++++++++++++++++++++++++------- 1 file changed, 52 insertions(+), 10 deletions(-) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index 28b1271..dd99564 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -2,13 +2,15 @@ You can create a file `/etc/pihole/pihole-FTL.conf` that will be read by *FTL*DN Possible settings (**the option shown first is the default**): -On which port should FTL be listening? - ## DNS settings ### BLOCKINGMODE `BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN` +How should `FTL` reply to blocked queries? + +**[More details](blockingmode.md)** + ## Statistics settings ### MAXLOGAGE @@ -38,10 +40,6 @@ Allow `FTL` to analyze AAAA queries from pihole.log? Should `FTL` only analyze A and AAAA queries? -How should `FTL` reply to blocked queries? - -**[More details](blockingmode.md)** - ## Socket settings ### SOCKET_LISTENING @@ -52,6 +50,8 @@ Listen only for local socket connections or permit all connections ### FTLPORT `FTLPORT=4711` +On which port should FTL be listening? + ## Host name resolution ### RESOLVE_IPV6 @@ -96,15 +96,57 @@ Specify path and filename of FTL's SQLite3 long-term database. Setting this to ` **[More details](database.md)** ## Debugging options +### DEBUG_DATABASE +`DEBUG_DATABASE=false|true` -### REGEX_DEBUGMODE -``` -REGEX_DEBUGMODE=false|true -``` +Print debugging information about database actions. This prints performed SQL statements as well as some general information such as the time it took to store the queries and how many have been saved to the database. + +### DEBUG_NETWORKING +`DEBUG_NETWORKING=false|true` + +Prints a list of the detected interfaces on startup of `pihole-FTL`. Also prints whether these interfaces are IPv4 or IPv6 interfaces. + +### DEBUG_LOCKS +`DEBUG_LOCKS=false|true` + +Print information about shared memory locks. Messages will be generated when waiting, obtaining, and releasing a lock. + +### DEBUG_QUERIES +`DEBUG_QUERIES=false|true` + +Print extensive query information (domains, types, replies, etc.). This has always been part of the legacy `debug` mode of `pihole-FTL`. + +### DEBUG_FLAGS +`DEBUG_FLAGS=false|true` + +Print flags of queries received by the DNS hooks. Only effective when `DEBUG_QUERIES` is enabled as well. + +### DEBUG_SHMEM +`DEBUG_SHMEM=false|true` + +Print information about shared memory buffers. Messages are either about creating or enlarging shmem objects or string injections. + +### DEBUG_GC +`DEBUG_GC=false|true` + +Print information about garbage collection (GC): What is to be removed, how many have been removed and how long did GC take. + +### DEBUG_ARP +`DEBUG_ARP=false|true` + +Print information about ARP table processing: How long did parsing take, whether read MAC addresses are valid, and if the `macvendor.db` file exists. + +### DEBUG_REGEX +`DEBUG_REGEX=false|true` Controls if *FTL*DNS should print extended details about regex matching into `pihole-FTL.log`. **[More details](regex/overview.md)** +Due to legacy reasons, we also support the following setting to be used for enabling the same functionality: + +`REGEX_DEBUGMODE=false|true` + +Note that if one of them is set to `true`, the other one cannot be used to disable this setting again. {!abbreviations.md!} From d597a687e9730b3ef475182eb4ee656ab7ac6b24 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Wed, 16 Jan 2019 09:50:45 +0100 Subject: [PATCH 3/8] Update REGEX_DEBUGMODE -> DEBUG_REGEX on the Regex subpage Signed-off-by: DL6ER --- docs/ftldns/regex/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ftldns/regex/overview.md b/docs/ftldns/regex/overview.md index 16858f5..7190d8d 100644 --- a/docs/ftldns/regex/overview.md +++ b/docs/ftldns/regex/overview.md @@ -14,13 +14,13 @@ To tell *FTL*DNS to reload the list, either: ## Pi-hole Regex debugging mode To ease the usage of regular expression filters in *FTL*DNS, we offer a regex debugging mode. Set ``` -REGEX_DEBUGMODE=true +DEBUG_REGEX=true ``` in your `/etc/pihole/pihole-FTL.conf` and restart `pihole-FTL` to enable or disable this mode. Once the debugging mode is enabled, each match will be logged to `/var/log/pihole-FTL.log` in the following format: ``` -[2018-07-17 17:40:51.304] DEBUG: Regex in line 2 "((^)|(\.))twitter\." matches "whatever.twitter.com" +[2018-07-17 17:40:51.304] Regex in line 2 "((^)|(\.))twitter\." matches "whatever.twitter.com" ``` The given line number corresponds to the line in the file `/etc/pihole/regex.list`. From 02f4a885fe458b2efd42d3e41f1610521b6c2de2 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Wed, 16 Jan 2019 10:27:26 +0100 Subject: [PATCH 4/8] Condense configuration subpage further. This makes it also more readable Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 175 +++++++++++++------------------------- 1 file changed, 57 insertions(+), 118 deletions(-) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index dd99564..82f5c1a 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -1,152 +1,91 @@ You can create a file `/etc/pihole/pihole-FTL.conf` that will be read by *FTL*DNS on startup. Possible settings (**the option shown first is the default**): +### DNS settings -## DNS settings - -### BLOCKINGMODE -`BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN` - -How should `FTL` reply to blocked queries? - +- `BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN`
+ How should `FTL` reply to blocked queries?
**[More details](blockingmode.md)** -## Statistics settings +### Statistics settings -### MAXLOGAGE -`MAXLOGAGE=24.0` - -Up to how many hours of queries should be imported from the database and logs? Maximum is 744 (31 days) - -### PRIVACYLEVEL -`PRIVACYLEVEL=0|1|2|3|4` - -Which privacy level is used? +- `MAXLOGAGE=24.0`
+ Up to how many hours of queries should be imported from the database and logs? Maximum is 744 (31 days) +- `PRIVACYLEVEL=0|1|2|3|4`
+ Which privacy level is used?
**[More details](privacylevels.md)** -### IGNORE_LOCALHOST -`IGNORE_LOCALHOST=no|yes` +- `IGNORE_LOCALHOST=no|yes`
+ Should `FTL` ignore queries coming from the local machine? -Should `FTL` ignore queries coming from the local machine? +- `AAAA_QUERY_ANALYSIS=yes|no`
+ Allow `FTL` to analyze AAAA queries from pihole.log? -### AAAA_QUERY_ANALYSIS -`AAAA_QUERY_ANALYSIS=yes|no` +- `ANALYZE_ONLY_A_AND_AAAA=false|true`
+ Should `FTL` only analyze A and AAAA queries? -Allow `FTL` to analyze AAAA queries from pihole.log? +### Socket settings -### ANALYZE_ONLY_A_AND_AAAA -`ANALYZE_ONLY_A_AND_AAAA=false|true` +- `SOCKET_LISTENING=localonly|all`
+ Listen only for local socket connections or permit all connections -Should `FTL` only analyze A and AAAA queries? +- `FTLPORT=4711`
+ On which port should FTL be listening? -## Socket settings +### Host name resolution -### SOCKET_LISTENING -`SOCKET_LISTENING=localonly|all` +- `RESOLVE_IPV6=yes|no`
+ Should `FTL` try to resolve IPv6 addresses to host names? -Listen only for local socket connections or permit all connections +- `RESOLVE_IPV4=yes|no`
+ Should `FTL` try to resolve IPv4 addresses to host names? -### FTLPORT -`FTLPORT=4711` +### Database settings +**[Further details concerning the database](database.md)** -On which port should FTL be listening? +- `DBIMPORT=yes|no`
+ Should `FTL` load information from the database on startup to be aware of the most recent history? -## Host name resolution +- `MAXDBDAYS=365`
+ How long should queries be stored in the database? Setting this to `0` disables the database -### RESOLVE_IPV6 -`RESOLVE_IPV6=yes|no` +- `DBINTERVAL=1.0`
+ How often do we store queries in FTL's database [minutes]? -Should `FTL` try to resolve IPv6 addresses to host names? +- `DBFILE=/etc/pihole/pihole-FTL.db`
+ Specify path and filename of FTL's SQLite3 long-term database. Setting `DBFILE=` disables the database altogether -### RESOLVE_IPV4 -`RESOLVE_IPV4=yes|no` +### Debugging options +- `DEBUG_DATABASE=false|true`
+ Print debugging information about database actions. This prints performed SQL statements as well as some general information such as the time it took to store the queries and how many have been saved to the database. -Should `FTL` try to resolve IPv4 addresses to host names? +- `DEBUG_NETWORKING=false|true`
+ Prints a list of the detected interfaces on startup of `pihole-FTL`. Also prints whether these interfaces are IPv4 or IPv6 interfaces. -## Database settings +- `DEBUG_LOCKS=false|true`
+ Print information about shared memory locks. Messages will be generated when waiting, obtaining, and releasing a lock. -### DBIMPORT -`DBIMPORT=yes|no` +- `DEBUG_QUERIES=false|true`
+ Print extensive query information (domains, types, replies, etc.). This has always been part of the legacy `debug` mode of `pihole-FTL`. -Should `FTL` load information from the database on startup to be aware of the most recent history? +- `DEBUG_FLAGS=false|true`
+ Print flags of queries received by the DNS hooks. Only effective when `DEBUG_QUERIES` is enabled as well. -**[More details](database.md)** +- `DEBUG_SHMEM=false|true`
+ Print information about shared memory buffers. Messages are either about creating or enlarging shmem objects or string injections. -### MAXDBDAYS -`MAXDBDAYS=365` +- `DEBUG_GC=false|true`
+ Print information about garbage collection (GC): What is to be removed, how many have been removed and how long did GC take. -How long should queries be stored in the database? -Setting this to `0` disables the database +- `DEBUG_ARP=false|true`
+ Print information about ARP table processing: How long did parsing take, whether read MAC addresses are valid, and if the `macvendor.db` file exists. -**[More details](database.md)** - -### DBINTERVAL -`DBINTERVAL=1.0` - -How often do we store queries in FTL's database [minutes]? - -**[More details](database.md)** - -### DBFILE -`DBFILE=/etc/pihole/pihole-FTL.db` - -Specify path and filename of FTL's SQLite3 long-term database. Setting this to `DBFILE=` disables the database altogether - -**[More details](database.md)** - -## Debugging options -### DEBUG_DATABASE -`DEBUG_DATABASE=false|true` - -Print debugging information about database actions. This prints performed SQL statements as well as some general information such as the time it took to store the queries and how many have been saved to the database. - -### DEBUG_NETWORKING -`DEBUG_NETWORKING=false|true` - -Prints a list of the detected interfaces on startup of `pihole-FTL`. Also prints whether these interfaces are IPv4 or IPv6 interfaces. - -### DEBUG_LOCKS -`DEBUG_LOCKS=false|true` - -Print information about shared memory locks. Messages will be generated when waiting, obtaining, and releasing a lock. - -### DEBUG_QUERIES -`DEBUG_QUERIES=false|true` - -Print extensive query information (domains, types, replies, etc.). This has always been part of the legacy `debug` mode of `pihole-FTL`. - -### DEBUG_FLAGS -`DEBUG_FLAGS=false|true` - -Print flags of queries received by the DNS hooks. Only effective when `DEBUG_QUERIES` is enabled as well. - -### DEBUG_SHMEM -`DEBUG_SHMEM=false|true` - -Print information about shared memory buffers. Messages are either about creating or enlarging shmem objects or string injections. - -### DEBUG_GC -`DEBUG_GC=false|true` - -Print information about garbage collection (GC): What is to be removed, how many have been removed and how long did GC take. - -### DEBUG_ARP -`DEBUG_ARP=false|true` - -Print information about ARP table processing: How long did parsing take, whether read MAC addresses are valid, and if the `macvendor.db` file exists. - -### DEBUG_REGEX -`DEBUG_REGEX=false|true` - -Controls if *FTL*DNS should print extended details about regex matching into `pihole-FTL.log`. - -**[More details](regex/overview.md)** - -Due to legacy reasons, we also support the following setting to be used for enabling the same functionality: - -`REGEX_DEBUGMODE=false|true` - -Note that if one of them is set to `true`, the other one cannot be used to disable this setting again. +- `DEBUG_REGEX=false|true`
+ Controls if *FTL*DNS should print extended details about regex matching into `pihole-FTL.log`.
+ **[More details](regex/overview.md)**
+ Due to legacy reasons, we also support the following setting to be used for enabling the same functionality:
+ `REGEX_DEBUGMODE=false|true`
+ Note that if one of them is set to `true`, the other one cannot be used to disable this setting again. {!abbreviations.md!} From a16eb51e1aa93f3ef95b784a5d1c65fcb9d2c6a4 Mon Sep 17 00:00:00 2001 From: Mcat12 Date: Mon, 11 Mar 2019 20:05:28 -0700 Subject: [PATCH 5/8] Add DEBUG_ALL, DEBUG_API, and DEBUG_OVERTIME Signed-off-by: Mcat12 --- docs/ftldns/configfile.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index 15d0e3d..211a8cf 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -58,6 +58,9 @@ Possible settings (**the option shown first is the default**): ### Debugging options +- `DEBUG_ALL=false|true`
+ Enable all debug flags. If this is set to true, all other debug config options are ignored. + - `DEBUG_DATABASE=false|true`
Print debugging information about database actions. This prints performed SQL statements as well as some general information such as the time it took to store the queries and how many have been saved to the database. @@ -89,4 +92,10 @@ Possible settings (**the option shown first is the default**): Note that if one of them is set to `true`, the other one cannot be used to disable this setting again.
**[More details](regex/overview.md)** +- `DEBUG_API=false|true`
+ Print extra debugging information during telnet API calls. Currently only used to send extra information when getting all queries. + +- `DEBUG_OVERTIME=false|true`
+ Print information about overTime memory operations, such as initializing or moving overTime slots. + {!abbreviations.md!} From 9804a092d4d4eb168d4d99465f9a654bd99cfb05 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Sat, 16 Mar 2019 16:53:33 +0100 Subject: [PATCH 6/8] Add description of DEBUG_EXTBLOCKED config option. Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index eb23a89..c7aa4c1 100755 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -97,6 +97,9 @@ How should `FTL` reply to blocked queries?
#### `DEBUG_OVERTIME=false|true` {#debug_overtime data-toc-label='Debug overTime'} Print information about overTime memory operations, such as initializing or moving overTime slots. +
+#### `DEBUG_EXTBLOCKED=false|true` {#debug_extblocked data-toc-label='Debug externally blocked'} + Print information about why FTL decided that certain queries were recognized as being externally blocked. {!abbreviations.md!} From 65311a51b7f9c6fc90e0f6f7afddb51a586ceda0 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Mon, 18 Mar 2019 07:46:10 +0100 Subject: [PATCH 7/8] Add description of DEBUG_CAPS config option. Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index c7aa4c1..e14c510 100755 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -100,6 +100,9 @@ How should `FTL` reply to blocked queries?
#### `DEBUG_EXTBLOCKED=false|true` {#debug_extblocked data-toc-label='Debug externally blocked'} Print information about why FTL decided that certain queries were recognized as being externally blocked. +
+#### `DEBUG_CAPS=false|true` {#debug_caps data-toc-label='Debug Linux capabilities'} + Print information about capabilities granted to the pihole-FTL process. The current capabilities are printed on receipt of `SIGHUP`, i.e., the current set of capabilities can be queried without restarting `pihole-FTL` (by setting `DEBUG_CAPS=true` and thereafter sending `killall -HUP pihole-FTL`). {!abbreviations.md!} From 891053e6a9f7036b84b805cfa798be231c4948c8 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Fri, 12 Apr 2019 16:28:53 +0200 Subject: [PATCH 8/8] Add documentation for finer-grained externally blocked status types. Signed-off-by: DL6ER --- docs/ftldns/database.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/ftldns/database.md b/docs/ftldns/database.md index 2210646..35176c7 100644 --- a/docs/ftldns/database.md +++ b/docs/ftldns/database.md @@ -101,6 +101,9 @@ ID | Query Type 3 | Permitted + replied to from cache 4 | Blocked by wildcard 5 | Blocked by `black.list` +6 | Blocked by upstream server (known blocking page IP address) +7 | Blocked by upstream server (`0.0.0.0` or `::`) +8 | Blocked by upstream server (`NXDOMAIN` with `RA` bit unset) ### Example for interaction with the FTL long-term database In addition to the interactions the Pi-hole database API offers, you can also run your own SQL commands against the database. If you want to obtain the three most queries domains for all time, you could use