From 551df17e8f1afef245246eadc7216de0d0cd9008 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Sun, 16 Jan 2022 21:44:32 +0100 Subject: [PATCH 1/8] Documentation update corresponding to https://github.com/pi-hole/FTL/pull/1255 Signed-off-by: DL6ER --- docs/database/ftl.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/database/ftl.md b/docs/database/ftl.md index 1d063e3..75cd741 100644 --- a/docs/database/ftl.md +++ b/docs/database/ftl.md @@ -51,6 +51,8 @@ Label | Type | Allowed to by empty | Content `forward` | text | Yes | Forward destination used for this query (only set if `status == 2`) `additional_info` | blob | Yes | Data-dependent content, see below +The `queries` `VIEW` is dynamically generated from the data actually stored in the tables `queries_storage` and the linking tables `domain_by_id`, `client_by_id`, `forward_by_id`, and `addinfo_by_id` (see below). The table `queries_storage` will contains integer IDs pointing to the respective entries of the linking tables to save space and make searching the database faster. If you haven't upgraded for some time, the table may still contain strings instead of integer IDs. + #### Data-dependent `additional_info` field The content and type of the `additional_info` row depends on the status of the given query. @@ -131,6 +133,45 @@ ID | Status | | Details 13 | Allowed | ✅ | Retried but ignored query (this may happen during ongoing DNSSEC validation) 14 | Allowed | ✅ | Already forwarded, not forwarding again +### Linking tables + +The `queries` `VIEW` reads repeating properties from linked tables to reduce both database size and search complexity. These linking tables, `domain_by_id`, `client_by_id`, `forward_by_id`, and `addinfo_by_id` all have a similar structure: + +#### `domain_by_id` + +Label | Type | Allowed to by empty | Content +--- | --- | --- | --- +`id` | integer | No | ID of the entry. Used by `query_storage` +`domain` | text | No | Domain name + +#### `client_by_id` + +Label | Type | Allowed to by empty | Content +--- | --- | --- | --- +`id` | integer | No | ID of the entry. Used by `query_storage` +`ip` | text | No | Client IP address +`name` | text | Yes | Client host name + +#### `forward_by_id` + +Label | Type | Allowed to by empty | Content +--- | --- | --- | --- +`id` | integer | No | ID of the entry. Used by `query_storage` +`forward` | text | No | Upstream server identifier (`#`) + +#### `addinfo_by_id` + +Label | Type | Allowed to by empty | Content +--- | --- | --- | --- +`id` | integer | No | ID of the entry. Used by `query_storage` +`type` | integer | No | Type of the `content` field +`content` | blob | No | Type-dependent content + +Valid `type` IDs are currently + +- `ADDINFO_CNAME_DOMAIN = 1` if `content` is a CNAME pointer, and +- `ADDINFO_REGEX_ID = 2` if `content` is an integer ID pointing to a regular expression in `gravity.db` + ### Example for interaction with the long-term query database In addition to the interactions the Pi-hole database API offers, you can also run your own SQL commands against the database. If you want to obtain the three most queries domains for all time, you could use From bf9d22ff03c7977255cb95dc587550acf635773e Mon Sep 17 00:00:00 2001 From: DL6ER Date: Wed, 26 Jan 2022 02:09:56 +0100 Subject: [PATCH 2/8] Relax number of tables statement Signed-off-by: DL6ER --- docs/database/ftl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/database/ftl.md b/docs/database/ftl.md index 75cd741..2669595 100644 --- a/docs/database/ftl.md +++ b/docs/database/ftl.md @@ -36,7 +36,7 @@ will create `/home/pi/pihole-FTL.db.backup` which is a copy of your long-term da --- -The long-term database contains three tables: +The long-term database contains several tables: ### Query Table From dbf671de8dd997d2282107a69da73b20bb6be80d Mon Sep 17 00:00:00 2001 From: DL6ER Date: Sun, 30 Jan 2022 11:34:10 +0100 Subject: [PATCH 3/8] Add documentation for FTL#1285 Signed-off-by: DL6ER --- docs/database/ftl.md | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/docs/database/ftl.md b/docs/database/ftl.md index 2669595..dd7d102 100644 --- a/docs/database/ftl.md +++ b/docs/database/ftl.md @@ -50,6 +50,9 @@ Label | Type | Allowed to by empty | Content `client` | text | No | Requesting client (IP address) `forward` | text | Yes | Forward destination used for this query (only set if `status == 2`) `additional_info` | blob | Yes | Data-dependent content, see below +`reply_type` | integer | Yes | Type of the reply for this query (see [Supported reply types](ftl.md#supported-reply-types)) +`reply_time` | real | Yes | Seconds it took until the final reply was received +`dnssec` | integer | Yes | Type of the DNSSEC status for this query (see [DNSSEC status](ftl.md#dnssec-status)) The `queries` `VIEW` is dynamically generated from the data actually stored in the tables `queries_storage` and the linking tables `domain_by_id`, `client_by_id`, `forward_by_id`, and `addinfo_by_id` (see below). The table `queries_storage` will contains integer IDs pointing to the respective entries of the linking tables to save space and make searching the database faster. If you haven't upgraded for some time, the table may still contain strings instead of integer IDs. @@ -108,11 +111,6 @@ ID | Resource Record (a.k.a. query type) 15 | `SVCB` 16 | `HTTPS` - - - - - ### Supported status types ID | Status | | Details @@ -133,6 +131,35 @@ ID | Status | | Details 13 | Allowed | ✅ | Retried but ignored query (this may happen during ongoing DNSSEC validation) 14 | Allowed | ✅ | Already forwarded, not forwarding again +### Supported reply types + +ID | Reply type is +--- | --- +0 | *unknown* (no reply so far) +1 | `NODATA` +2 | `NXDOMAIN` +3 | `CNAME` +4 | a valid `IP` record +5 | `DOMAIN` +6 | `RRNAME` +7 | `SERVFAIL` +8 | `REFUSED` +9 | `NOTIMP` +10 | `OTHER` +11 | `DNSSEC` +12 | `NONE` (query was dropped intentionally) +13 | `BLOB` (binary data) + +### DNSSEC status + +ID | DNSSEC status is +--- | --- +0 | *unknown* +1 | `SECURE` +2 | `INSECURE` +3 | `BOGUS` +4 | `ABANDONED` + ### Linking tables The `queries` `VIEW` reads repeating properties from linked tables to reduce both database size and search complexity. These linking tables, `domain_by_id`, `client_by_id`, `forward_by_id`, and `addinfo_by_id` all have a similar structure: From 242559ff9b83cf01f850bd27ee0f74a2e13f2fd9 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Sat, 5 Feb 2022 10:45:41 +0100 Subject: [PATCH 4/8] Add documentation for https://github.com/pi-hole/FTL/pull/1295 Signed-off-by: DL6ER --- docs/ftldns/telnet-api.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/ftldns/telnet-api.md b/docs/ftldns/telnet-api.md index 0347bbb..893139a 100644 --- a/docs/ftldns/telnet-api.md +++ b/docs/ftldns/telnet-api.md @@ -161,6 +161,12 @@ Connect via e.g. `telnet 127.0.0.1 4711` or use `echo ">command" | nc 127.0.0.1 53 ``` +- `>maxlogage`: Get timespan of the statistics shown on the dashboard + + ```text + 86400 + ``` + Note that the port can also be `0` if someone decides to disable the DNS server part of Pi-hole {!abbreviations.md!} From 9d78046fd048f8a08e6cd71dfe74734f3426e7d7 Mon Sep 17 00:00:00 2001 From: DL6ER Date: Sat, 5 Feb 2022 11:07:10 +0100 Subject: [PATCH 5/8] Add documentation for new options LOCAL_IPV4/6 and BLOCK_IPV4/6. Deprecate options REPLY_ADDR4/6. Signed-off-by: DL6ER --- docs/ftldns/configfile.md | 41 +++++++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index 75eee83..9b230e5 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -42,23 +42,29 @@ For this setting, both numbers, the maximum number of queries within a given tim Rate-limiting may be disabled altogether by setting `RATE_LIMIT=0/0` (this results in the same behavior as before FTL v5.7). -#### `REPLY_ADDR4=` (unset by default, PR [#965](https://github.com/pi-hole/FTL/pull/965)) {#reply_addr4 data-toc-label='Force A reply'} +#### `LOCAL_IPV4=` (unset by default, PR [#1293](https://github.com/pi-hole/FTL/pull/1293)) {#local_ipv4 data-toc-label='Force local A reply'} -`FTL` determines the address of the interface a query arrived on. We then use this IP address when replying to queries with an `A` record (IPv4 address). This setting can be used to ensure a fixed, rather than the dynamically obtained, address is used. +By default, `FTL` determines the address of the interface a query arrived on and uses this address for replying to `A` queries with the most suitable address for the requesting client. This setting can be used to use a fixed, rather than the dynamically obtained, address when Pi-hole responds to the following names: -This applies to the following cases: +- `pi.hole` +- `` +- `pi.hole.` +- `.` + +#### `LOCAL_IPV6=` (unset by default, PR [#1293](https://github.com/pi-hole/FTL/pull/1293)) {#local_ipv6 data-toc-label='Force local AAAA reply'} + +Used to overwrite the IP address for local `AAAA` queries. See [`LOCAL_IPV4`](#local_ipv4) for details when this setting is used. + +#### `BLOCK_IPV4=` (unset by default, PR [#1293](https://github.com/pi-hole/FTL/pull/1293)) {#block_ipv4 data-toc-label='Force blocked A reply'} + +By default, `FTL` determines the address of the interface a query arrived on and uses this address for replying to `A` queries with the most suitable address for the requesting client. This setting can be used to use a fixed, rather than the dynamically obtained, address when Pi-hole responds in the following cases: - `IP` blocking mode is used and this query is to be blocked - A regular expression with the [`;reply=IP` regex extension](../regex/pi-hole.md#specify-reply-type) is used -- Pi-hole responds to one of the following names - - `pi.hole` - - `` - - `pi.hole.` - - `.` -#### `REPLY_ADDR6=` (unset by default, PR [#965](https://github.com/pi-hole/FTL/pull/965)) {#reply_addr6 data-toc-label='Force AAAA reply'} +#### `BLOCK_IPV6=` (unset by default, PR [#1293](https://github.com/pi-hole/FTL/pull/1293)) {#block_ipv6 data-toc-label='Force blocked AAAA reply'} -`FTL` determines the address of the interface a query arrived on. We then use this IP address when replying to queries with an `AAAA` record (IPv6 address). This setting can be used to ensure a fixed, rather than the dynamically obtained, address is used. See [`REPLY_ADDR4`](#reply_addr4) for details about when this setting is used. +Used to overwrite the IP address for blocked `AAAA` queries. See [`BLOCK_IPV4`](#block_ipv4) for details when this setting is used. #### `REPLY_WHEN_BUSY=ALLOW|DROP|BLOCK|REFUSE` (PR [#1156](https://github.com/pi-hole/FTL/pull/1156)) {#reply_when_busy data-toc-label='Database busy reply'} @@ -377,4 +383,19 @@ Should FTL translate its own stack addresses into code lines during the bug back Temporary flag that may print additional information. This debug flag is meant to be used whenever needed for temporary investigations. The logged content may change without further notice at any time. +### Deprecated options + +#### `REPLY_ADDR4=` (unset by default, PR [#965](https://github.com/pi-hole/FTL/pull/965)) {#reply_addr4 data-toc-label='Force A reply'} + +*This option is deprecated and may be removed in future versions, please use `BLOCK_IPV4` and `LOCAL_IPV4` instead* + +If neither `BLOCK_IPV4` nor `LOCAL_IPV4` are set, this setting use used to set both of them. If any of the two is set, this setting is ignored altogether. + +#### `REPLY_ADDR6=` (unset by default, PR [#965](https://github.com/pi-hole/FTL/pull/965)) {#reply_addr6 data-toc-label='Force AAAA reply'} + +*This option is deprecated and may be removed in future versions, please use `BLOCK_IPV6` and `LOCAL_IPV6` instead* + +If neither `BLOCK_IPV4` nor `LOCAL_IPV4` are set, this setting use used to set both of them. If any of the two is set, this setting is ignored altogether. + + {!abbreviations.md!} From d1558679471701abce37f00d6bbd3e9dd90f3006 Mon Sep 17 00:00:00 2001 From: yubiuser Date: Sat, 12 Feb 2022 22:46:05 +0100 Subject: [PATCH 6/8] Add seconds --- docs/ftldns/telnet-api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ftldns/telnet-api.md b/docs/ftldns/telnet-api.md index 893139a..0df2784 100644 --- a/docs/ftldns/telnet-api.md +++ b/docs/ftldns/telnet-api.md @@ -161,7 +161,7 @@ Connect via e.g. `telnet 127.0.0.1 4711` or use `echo ">command" | nc 127.0.0.1 53 ``` -- `>maxlogage`: Get timespan of the statistics shown on the dashboard +- `>maxlogage`: Get timespan of the statistics shown on the dashboard (in seconds) ```text 86400 From 995718a0f10053afc3369f83acb451e78e9adb95 Mon Sep 17 00:00:00 2001 From: yubiuser Date: Sat, 12 Feb 2022 22:52:18 +0100 Subject: [PATCH 7/8] Apply suggestions from code review --- docs/ftldns/configfile.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ftldns/configfile.md b/docs/ftldns/configfile.md index 9b230e5..7df1d6d 100644 --- a/docs/ftldns/configfile.md +++ b/docs/ftldns/configfile.md @@ -389,13 +389,13 @@ Temporary flag that may print additional information. This debug flag is meant t *This option is deprecated and may be removed in future versions, please use `BLOCK_IPV4` and `LOCAL_IPV4` instead* -If neither `BLOCK_IPV4` nor `LOCAL_IPV4` are set, this setting use used to set both of them. If any of the two is set, this setting is ignored altogether. +If neither `BLOCK_IPV4` nor `LOCAL_IPV4` are set, this setting is used to set both of them. If either of the two is set, this setting is ignored altogether. #### `REPLY_ADDR6=` (unset by default, PR [#965](https://github.com/pi-hole/FTL/pull/965)) {#reply_addr6 data-toc-label='Force AAAA reply'} *This option is deprecated and may be removed in future versions, please use `BLOCK_IPV6` and `LOCAL_IPV6` instead* -If neither `BLOCK_IPV4` nor `LOCAL_IPV4` are set, this setting use used to set both of them. If any of the two is set, this setting is ignored altogether. +If neither `BLOCK_IPV6` nor `LOCAL_IPV6` are set, this setting is used to set both of them. If either of the two is set, this setting is ignored altogether. {!abbreviations.md!} From 3f24484e72236f4f1b36e46911bb71d15e5e3cce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Christian=20K=C3=B6nig?= Date: Sat, 12 Feb 2022 23:09:21 +0100 Subject: [PATCH 8/8] Adjust blockingmode.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Christian König --- docs/ftldns/blockingmode.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ftldns/blockingmode.md b/docs/ftldns/blockingmode.md index 2de543d..e6be9fc 100644 --- a/docs/ftldns/blockingmode.md +++ b/docs/ftldns/blockingmode.md @@ -47,7 +47,7 @@ Following [RFC 3513, Internet Protocol Version 6 (IPv6) Addressing Architecture, BLOCKINGMODE=IP-NODATA-AAAA ``` -Blocked queries will be answered with the local IPv4 addresses of your Pi-hole (as configured in your `setupVars.conf` file). Blocked AAAA queries will be answered with `NODATA-IPV6` and clients will only try to reach your Pi-hole over its static IPv4 address +Blocked queries will be answered with the local IPv4 addresses of your Pi-hole (see [BLOCK_IP4](configfile.md#block_ipv4) for additional options). Blocked AAAA queries will be answered with `NODATA-IPV6` and clients will only try to reach your Pi-hole over its static IPv4 address ``` ;; QUESTION SECTION: @@ -78,7 +78,7 @@ doubleclick.net. 2 IN A 192.168.2.11 BLOCKINGMODE=IP ``` -Blocked queries will be answered with the local IP addresses of your Pi-hole (as configured in your `setupVars.conf` file) +Blocked queries will be answered with the local IP addresses of your Pi-hole (see [BLOCK_IP4](configfile.md#block_ipv4) and [BLOCK_IP6](configfile.md#block_ipv6) for additional options) ``` ;; QUESTION SECTION: