Improve regex documentation

Signed-off-by: DL6ER <dl6er@dl6er.de>
This commit is contained in:
DL6ER
2021-02-02 10:02:08 +01:00
parent 6293accf1b
commit fcee2c01e5
11 changed files with 300 additions and 28 deletions
+97
View File
@@ -0,0 +1,97 @@
# Approximative matching
You may or not be know `agrep`. It is basically a "forgiving" `grep` and is, for instance, used for searching through (offline) dictionaries. It is tolerant against errors (up to degree you specify). It may be beneficial is you want to match against domains where you don't really know the pattern. It is just an idea, we will have to see if it is actually useful.
This is a somewhat complicated topic, we'll approach it by examples as it is very complicated to get the head around it by just listening to the specifications.
The approximate matching settings for a subpattern can be changed by appending *approx-settings* to the subpattern. Limits for the number of errors can be set and an expression for specifying and limiting the costs can be given:
## Accepted **insertions** (`+`)
Use `(something){+x}` to specify that the regex should still be matching when `x` characters would need it be *inserted* into the sub-expression `something`.
Example:
- `doubleclick.net` is matched by `^doubleclick\.(nt){+1}$`
The missing `e` in `nt` is inserted.
Similarly:
- `doubleclick.net` is matched by `^(doubleclk\.nt){+3}$`
The missing characters in the domain are substituted. The maximum number of insertions spans the entire domain as is wrapped in the sub-expression `(...)`.
## Accepted **deletions** (`-`)
Use `(something){-x}` to specify that the regex should still be matching when `x` characters would need it be *deleted* from the sub-expression `something`:
Example:
- `doubleclick.net` is matched by `^doubleclick\.(neet){-1}$`
The surplus `e` in `neet` is deleted.
Similarly:
- `doubleclick.net` is matched by `^(doubleclicky\.netty){-3}$`
- `doubleclick.net` is NOT matched by `^(doubleclicky\.nettfy){-3}$`
## Accepted **substitutions** (`#`)
Use `(something){#x}` to specify that the regex should still be matching when `x` characters would need to be *substituted* from the sub-expression `something`:
Example 1:
- `oobargoobaploowap` is matched by `(foobar){#2~2}`
Hint: `goobap` is `foobar` with two substitutions `f->g` and `r->p`
Example 2:
- `doubleclick.net` is matched by `^doubleclick\.n(tt){#1}$`
The incorrect `t` in `ntt` is substituted. Note that substitutions are necessary when a character needs to be replaced as the corresponding realization with one insertion and one deletion is **not identical**:
`doubleclick.net` is matched by `^doubleclick\.n(tt){+1-1}$`
(`t` is removed, `e` is added), however
- `doubleclick.nt` is ALSO matched by `^doubleclick\.n(tt){+1-1}$`
(the `t` is just removed, nothing had to be added) but
- `doubleclick.nt` is NOT matched by `^doubleclick\.n(tt){#1}$`
doesn't match as substitutions always require characters to be swapped by others.
## Combinations and total error limit (`~`)
All rules from above can be combined like as `{+2-5#6}` allowing (up to!) two insertions, five deletions, and six substitutions. You can enforce an upper limit on the number of tried realizations using the tilde. Even when `{+2-5#6}` can lead to up to 13 operations being tried, this can be limited to (at most) seven tries using `{+2-5#6~7}`.
Example:
- `oobargoobploowap` is matched by `(foobar){+2#2~3}`
Hint: `goobaap` is `foobar` with
- two substitutions `f->g` and `r->p`, and
- one addition `a` between `bar` (to have `baap`)
Specifying `~2` instead of `~3` will lead to no match as three errors need to be corrected in total for a match in this example.
## Advanced topic: Cost-equation
You can even weight the "costs" of insertions, deletions or substitutions. This is really an advanced topic and should only be touched when really needed.
A *cost-equation* can be thought of as a mathematical equation, where `i`, `d`, and `s` stand for the number of insertions, deletions, and substitutions, respectively. The equation can have a multiplier for each of `i`, `d`, and `s`.
The multiplier is the **cost of the error**, and the number after `<` is the maximum allowed total cost of a match. Spaces and pluses can be inserted to make the equation more readable. When specifying only a cost equation, adding a space after the opening `{` is **required** .
Example 1: `{ 2i + 1d + 2s < 5 }`
This sets the cost of an insertion to two, a deletion to one, a substitution to two, and the maximum cost to five.
Example 2: `{+2-5#6, 2i + 1d + 2s < 5 }`
This sets the cost of an insertion to two, a deletion to one, a substitution to two, and the maximum cost to five. Furthermore, it allows only up to 2 insertions (coming at a total cost of 4), five deletions and up to 6 substitutions. As six substitutions would come at a cost of `6*2 = 12`, exeeding the total allowed costs of 5, they cannot all be realized.
{!abbreviations.md!}
+45
View File
@@ -0,0 +1,45 @@
A regular expression, or RegEx for short, is a pattern that **can be used for building arbitrarily complex filter** rules in *FTL*DNS.
We implement the POSIX Extended Regular Expressions similar to the one used by the UNIX `egrep` (or `grep -E`) command. We amend the regex engine by approximate blocking (compare to `agrep`) and other special features like matching to specific query types only.
Our implementation is light and fast as each domain is only checked once for a match. When you query `google.com`, it will be checked against your RegEx. Any subsequent query to the same domain will not be checked again until you restart `pihole-FTL`.
## Hierarchy of regex filters in *FTL*DNS
*FTL*DNS uses a specific hierarchy to ensure regex filters work as you expect them to. Whitelisting always has priority over blacklisting.
There are two locations where regex filters are important:
1. On loading the blocking domains form the `gravity` database table, *FTL*DNS skips not only exactly whitelisted domains but also those that match enabled whitelist regex filters.
2. When a queried domain matches a blacklist regex filter, the query will *not* be blocked if the domain *also* matches an exact or a regex whitelist entry.
## How to use regular expressions for filtering domains
*FTL*DNS reads in regular expression filters from the two [`regex` database views](../database/gravity/index.md).
To tell *FTL*DNS to reload the list of regex filters, either:
- Execute `pihole restartdns reload-lists` or
- Send `SIGHUP` to `pihole-FTL` (`sudo killall -SIGHUP pihole-FTL`) or
- Restart the service (`sudo service pihole-FTL restart`)
The first command is to be preferred as it ensures that the DNS cache itself remains intact. Hence, it is also the fastest of the available options.
## Pi-hole Regex debugging mode
To ease the usage of regular expression filters in *FTL*DNS, we offer a regex debugging mode. Set
``` plain
DEBUG_REGEX=true
```
in your `/etc/pihole/pihole-FTL.conf` and restart `pihole-FTL` to enable or disable this mode.
Once the debugging mode is enabled, each match will be logged to `/var/log/pihole-FTL.log` in the following format:
```text
[2018-07-17 17:40:51.304] Regex blacklist (DB ID 15) >> MATCH: "whatever.twitter.com" vs. "((^)|(\.))twitter\."
```
The given DB ID corresponds to the ID of the corresponding row in the `domainlist` database table.
Note that validation is only done on the first occurrence of a domain to increase the computational efficiency of *FTL*DNS. The result of this evaluation is stored in an internal DNS cache that is separate from `dnsmasq`'s own DNS cache. This allows us to only flush this special cache when modifying the black- and whitelists *without* having to flush the entire DNS cache collected so far.
{!abbreviations.md!}
+144
View File
@@ -0,0 +1,144 @@
# Pi-hole regex extensions
## Only match specific query types
You can amend the regular expressions by special keywords added at the end to fine-tine regular expressions to match only specific query types.
Example:
``` plain
abc;querytype=AAAA
```
will block
``` bash
dig AAAA abc
```
but not
``` bash
dig A abc
```
This allows you to do query type based black-/whitelisting. Some user-provided examples are:
- `.*;querytype=!A`
A regex blacklist entry for blocking `AAAA` (in fact, everything else than `A`, call it "anti-`A`") requests for all clients assigned to the same group. This has been mentioned to be benefitial for devices like Chromecast. You may want to fine-tune this further to specific domains.
- `.*;querytype=PTR`
A regex whitelist entry used to permit `PTR` lookups with the above "anti-`A`" regex
- `.*;querytype=ANY`
A regex blacklist entry to block `ANY` request network wide.
## Invert matching
Sometimes, it may be useful to be able to invert a regular expression altogether. Hence, we added the keyword `;invert` to achieve exactly this.
For instance,
``` plain
^abc$;querytype=AAAA;invert
```
will not block `abc` with type `AAAA` (but everything else) for the clients assigned to the same groups. This inversion is independent for the query type, e.g.
``` plain
^abc$;invert
```
will block **not** block `abc` but **everything else**.
## Comments
You can specify comments withing your regex using the syntax
``` plain
(?#some comment here)
```
The comment can contain any characters except for a closing parenthesis `)` (for the sole reason being the terminating element). The text in the comment is completely ignored by the regex parser and it used solely for readability purposes.
``` plain
$ pihole-FTL regex-test "doubleclick.net" "(^|\.)doubleclick\.(?#TODO: We need to maybe support more than just .net here)net$"
FTL Regex test:
Domain: "doubleclick.net"
Regex: "(^|\.)doubleclick\.(?#TODO: We need to maybe support more than just .net here)net$"
Step 1: Compiling regex filter...
Compiled regex filter in 0.167 msec
Step 2: Checking domain...
Done in 0.032 msec
MATCH
```
## Back-references
A back reference is a backslash followed by a single non-zero decimal digit `d`. It matches *the same sequence* of characters matched by the `d`th parenthesized subexpression.
Example:
``` plain
"cat.foo.dog---cat%dog!foo" is matched by "(cat)\.(foo)\.(dog)---\1%\3!\2"
```
Another (more complex example is):
``` plain
(1234|4321)\.(foo)\.(dog)--\1
```
``` plain
MATCH: 1234.foo.dog--1234
MATCH: 4321.foo.dog--4321
NO MATCH: 1234.foo.dog--4321
```
Mind that the last line gives no match as `\1` matches **exactly** the same sequence the first character group matched. And `4321` is not the same as `1234` even when both are valid replies for `(1234|4321)` Back references are not defined for POSIX EREs (for BREs they are, surprisingly enough). We add them to ERE in the BRE style.
``` plain
$ pihole-FTL regex-test "someverylongandmaybecomplexthing.foo.dog--someverylongandmaybecomplexthing" "(someverylongandmaybecomplexthing|somelesscomplexitem)\.(foo)\.(dog)--\1"
FTL Regex test:
Domain: "someverylongandmaybecomplexthing.foo.dog--someverylongandmaybecomplexthing"
Regex: "(someverylongandmaybecomplexthing|somelesscomplexitem)\.(foo)\.(dog)--\1"
Step 1: Compiling regex filter...
Compiled regex filter in 0.563 msec
Step 2: Checking domain...
Done in 0.031 msec
MATCH
```
## More character classes for bracket expressions
A bracket expression specifies a set of characters by enclosing a nonempty list of items in brackets. Normally anything matching any item in the list is matched. If the list begins with `^` the meaning is negated; any character matching no item in the list is matched.
1. Multiple characters: `[abc]` matches `a`, `b`, and `c`.
2. Character ranges: `[0-9]` matches any decimal digit.
3. Character classes:
- `[:alnum:]` alphanumeric characters
- `[:alpha:]` alphabetic characters
- `[:blank:]` blank characters
- `[:cntrl:]` control characters
- `[:digit:]` decimal digits (0 - 9)
- `[:graph:]` all printable characters except space
- `[:lower:]` lower-case letters (FTL matches case-insensitive by default)
- `[:print:]` printable characters including space
- `[:punct:]` printable characters not space or alphanumeric
- `[:space:]` white-space characters
- `[:upper:]` upper case letters (FTL matches case-insensitive by default)
- `[:xdigit:]` hexadecimal digits
Furthermore, there are two shortcurts for some character classes:
- `\d` - Digit character (equivalent to `[[:digit:]]`)
- `\D` - Non-digit character (equivalent to `[^[:digit:]]`)
{!abbreviations.md!}
+19
View File
@@ -0,0 +1,19 @@
# Regex Test mode
In order to ease regex development, we added a regex test mode to `pihole-FTL` which can be invoked like
``` bash
pihole-FTL regex-test doubleclick.net
```
(test `doubleclick.net` against all regexs in the gravity database), or
``` bash
pihole-FTL regex-test doubleclick.net "(^|\.)double"
```
(test `doubleclick.net` against the CLI-provided regex `(^|\.)double`.
You do NOT need to be `sudo` for this, any arbitrary user should be able to run this command. The test returns `0` on match and `1` on no match and errors, hence, it may be used for scripting.
{!abbreviations.md!}
+144
View File
@@ -0,0 +1,144 @@
# Pi-hole regular expressions tutorial
We provide a short but thorough introduction to our regular expressions implementation. This may come in handy if you are designing blocking or whitelisting rules (see also our cheat sheet below!). In our implementation, all characters match themselves except for the following special characters: `.[{}()\*+?|^$`. If you want to match those, you need to escape them like `\.` for a literal period, but no rule without exception (see character groups below for further details).
## Anchors (`^` and `$`)
First of all, we look at anchors that can be used to indicate the start or the end of a domain, respectively. If you don't specify anchors, the match may be partial (see examples below).
Example | Interpretation
--- | ---
`domain` | **partial match**. Without anchors, a text may appear anywhere in the domain. This matches `some.domain.com`, `domain.com` and `verylongdomain.com` and more
`^localhost$` | **exact match** matching *only* `localhost` but neither `a.localhost` nor `localhost.com`
`^abc` | matches any domain **starting** (`^`) in "abc" like `abcdomain.com`, `abc.domain.com` but not `def.abc.com`
`com$` | matches any domain **ending** (`$`) in "com" such as `domain.com` but not `domain.com.co.uk`
## Wildcard (`.`)
An unescaped period stands for any *single* character.
Example | Interpretation
--- | ---
`^domain.$` | matches `domaina`, `domainb`, `domainc`, but not `domain`
## Bounds and multipliers (`{}`, `*`, `+`, and `?`)
With bounds, one can denote the number of times something has to occur:
Bound | Meaning
--- | ---
`ab{4}` | matches a domain that contains a single `a` followed by four `b` (matching only `abbbb`)
`ab{4,}` | matches a domain that contains a single `a` followed by *at least* four `b` (matching also `abbbbbbbb`)
`ab{3,5}` | matches a domain that contains a single `a` followed by three to five `b` (matching only `abbb`, `abbbb`, and `abbbbb`)
Multipliers are shortcuts for some of the bounds that are needed most often:
Multipliers | Bounds equivalent | Meaning
--- | --- | ---
`?` | `{0,1}` | never or once (optional)
`*` | `{0,}` | never or more (optional)
`+` | `{1,}` | once or more (mandatory)
To illustrate the usefulness of multipliers (and bounds), we provide a few examples:
Example | Interpretation
--- | ---
`^r-*movie` | matches a domain like `r------movie.com` where the number of dashes can be arbitrary (also none)
`^r-?movie` | matches only the domains `rmovie.com` and `r-movie.com` but not those with more than one dash
`^r-+movie` | matches only the domains with at least one dash, i.e., not `rmovie.com`
`^a?b+` | matches domains like `abbbb.com` (zero or one `a` at the beginning followed by one or more `b`)
## Character groups (`[]`)
With character groups, a set of characters can be matched:
Character group | Interpretation
--- | ---
`[abc]` | matches `a`, `b`, or `c` (using explicitly specified characters)
`[a-c]` | matches `a`, `b`, or `c` (using a *range*)
`[a-c]+` | matches any non-zero number of `a`, `b`, `c`
`[a-z]` | matches any single lowercase letter
`[a-zA-Z]` | matches any single letter
`[a-z0-9]` | matches any single lowercase letter or any single digit
`[^a-z]` | **Negation** matching any single character *except* lowercase letters
`abc[0-9]+` | matches the string `abc` followed by a number of arbitrary length
Bracket expressions are an exception to the character escape rule. Inside them, all special characters, including the backslash (`\`), lose their special powers, i.e. they match themselves exactly. Furthermore, to include a literal `]` in the list, make it the first character (like `[]]` or `[^]]` if negated). To include a literal `-`, make it the first or last character, or the second endpoint of a range (e.g. `[a-z-]` to match `a` to `z` and `-`).
## Groups (`()`)
Using groups, we can enclose regular expressions, they are most powerful when combined with bounds or multipliers (see also alternations below).
Example | Interpretation
--- | ---
`(abc)` | matches `abc` (trivial example)
`(abc)*` | matches zero or more copies of `abc` like `abcabc` but not `abcdefabc`
`(abc){1,3}` | matches one, two or three copies of `abc`: `abc`, `abcabc`, `abcabcabc` but nothing else
## Alternations (`|`)
Alternations can be used as an "or" operator in regular expressions.
Example | Interpretation
--- | ---
`(abc)|(def)` | matches `abc` *and* `def`
`domain(a|b)\.com` | matches `domaina.com` and `domainb.com` but not `domain.com` or `domainx.com`
`domain(a|b)*\.com` | matches `domain.com`, `domainaaaa.com` `domainbbb.com` but not `domainab.com` (any number of `a` or `b` in between `domain` and `.com`)
## Character classes (`[:class:]`)
In addition to character groups, there are also some special character classes available, such as
Character class | Group equivalent | Pi-hole specific | Interpretation
--------------- | ---------------- | ---------------- | ---------------
`[:digit:]` | `[0-9]` | No | matches digits
`[:lower:]` | `[a-z]` | No | matched lowercase letters(FTL matches case-insensitive by default)
`[:upper:]` | `[A-Z]` | No | matched uppercase letters(FTL matches case-insensitive by default)
`[:alpha:]` | `[A-Za-z]` | No | matches alphabetic characters
`[:alnum:]` | `[A-Za-z0-9]` | No | matches alphabetic characters and digits
`[:blank:]` | `[ \t]` | Yes | blank characters
`[:cntrl:]` | N/A | Yes | control characters
`[:graph:]` | N/A | Yes | all printable characters except space
`[:print:]` | N/A | Yes | printable characters including space
`[:punct:]` | N/A | Yes | printable characters not space or alphanumeric
`[:space:]` | `[ \f\n\r\t\v]` | Yes | white-space characters
`[:xdigit:]` | `[0-9a-fA-F]` | Yes | hexadecimal digits
# Advanced examples
After going through our quick tutorial, we provide some more advanced examples so you can test your knowledge.
## Block domain with only numbers
```
^[0-9][^a-z]+\.((com)|(edu))$
```
Blocks domains containing only numbers (no letters) and ending in `.com` or `.edu`. This blocks `555661.com`, and `456.edu`, but not `555g555.com`
### Block domains without subdomains
```
^[a-z0-9]+([\-]{1}[a-z0-9]+)*\.[a-z]{2,7}$
```
A domain name shall not start or end with a dash but can contain any number of them. It must be followed by a TLD (we assume a valid TLD length of two to seven characters)
# Cheatsheet
Expression | Meaning | Example
------------ | ------------- | -----------
`^` | Beginning of string | `^client` matches strings that begin with `client`, such as `client.server.com` but not `more.client.server.com` (exception: within a character range (`[]`) `^` means negation)
`$` | End of string | `ing$` matches `exciting` but not `ingenious`
`*` | Match zero or more of the previous | `ah*` matches `ahhhhh` or `a`
`?` | Match zero or one of the previous | `ah?` matches `a` or `ah`
`+` | Match one or more of the previous | `ah+` matches `ah` or `ahhh` but not `a`
`.` | Wildcard character, matches any character | `do.*` matches `do`, `dog`, `door`, `dot`, etc.;<br>`do.+` matches `dog`, `door`, `dot`, etc. but not `do` (wildcard with `+` requires at least one extra character for matching)
`( )` | Group | Enclose regular expressions, see the example for `|`
`|` | Alternation | `(mon|tues)day` matches `monday` or `tuesday` but not `friday` or `mondiag`
`[ ]` | Matches a range of characters | `[cbf]ar` matches `car`, `bar`, or `far`;
`[^]`| Negation | `[^0-9]` matches any character *except* `0` to `9`
`{ }` | Matches a specified number of occurrences of the previous | `[0-9]{3}` matches any three-digit number like `315` but not `31`;<br>`[0-9]{2,4}` matches two- to four-digit numbers like `12`, `123`, and `1234` but not `1` or `12345`;<br>`[0-9]{2,}` matches any number with two or more digits like `1234567`, `123456789`, but not `1`
`\` | Used to escape a special character not inside `[]` | `google\.com` matches `google.com`
{!abbreviations.md!}