From 5cb47d4510ee48165af0e1b56ceaddea093af90d Mon Sep 17 00:00:00 2001 From: Daniel Valentine Date: Sat, 5 Nov 2022 20:43:58 -0600 Subject: [PATCH] Add instances.md and script for generating said. --- README.md | 34 ++++++- generate-instances-markdown.py | 178 +++++++++++++++++++++++++++++++++ instances.json | 5 - instances.md | 54 ++++++++++ instances.txt | 1 - 5 files changed, 264 insertions(+), 8 deletions(-) create mode 100755 generate-instances-markdown.py create mode 100644 instances.md diff --git a/README.md b/README.md index b3fc76b..0222d6e 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ This repo consists of four files: 1. `instances.json`: This is the list of Libreddit instances. 1. `instances-schema.json`: JSON Schema governing `instances.json`. 1. `instances.txt`: This is a CSV of libreddit instances. While this is also machine-readable, it is recommended to use `instances.json` instead. `instances.txt` is meant for contributors to add and remove instances, and `generate-instances-json.sh` will validate those instances and generate `instances.json`. +1. `instances.md`: This is table in Markdown format of the Libreddit instances in `instances.json`. It is generated by the script `generate-instances-markdown.py`. 1. `generate-instances-json.sh`: This script takes in a CSV file as input, typically `instances.txt`, and outputs a JSON object with a list of Libreddit instances. This is the script that generates `instances.json`. ## Adding or removing an instance @@ -17,12 +18,13 @@ To generate `instances.json`, perform the following: 1. Modify `instances.txt` to add or remove instances. See [Expected CSV format](#Expected CSV format) for the expected format of each CSV row. 1. Run `generate-instances-json.sh -i ./instances.txt -o ./instances.json` to generate `instances.json`. The existing `instances.json` will be replaced. +1. Run `generate-instances-markdown.py --output=./instances.md ./instances.json` to generate `instances.md`. The existing `instances.md` file will be replaced. Pull requests to add or remove instances are always welcome. ## `generate-instances-json.sh` -`generate-instances-json.sh` is the script that produces a JSON of [Libreddit](https://github.com/spikecodes/libreddit) instances, given a CSV input of Libreddit instances. +`generate-instances-json.sh` is the script that produces a JSON of [Libreddit](https://github.com/libreddit/libreddit) instances, given a CSV input of Libreddit instances. Unless `-i` and `-o` are specified (see [Usage](#Usage) below), the input and output are assumed the stdin and stdout streams respectively. @@ -131,6 +133,34 @@ If all instances could be processed, the script exits with an exit code of 0. If This script will attempt to connect to instances that are onion sites. To make sure it can do this, it will see if Tor is running and if torsocks is installed. If neither condition is met, the script will not attempt to connect to Libreddit onion sites and will skip them. The exit code will still be 0, assuming that the WWW Libreddit sites were processed without error. +## `generate-instances-markdown.py` + +`generate-instances-markdown.py` will generate a table in Markdown format of the instances. This requires the JSON file that is generated by `generate-instances-json.sh`. + +### Usage + +``` +usage: generate-instances-markdown.py [-h] [-o OUTPUT_FILE] [INPUT_FILE] + +Generate a markdown table of the Libreddit instances in the instances.JSON file. By default, this will read the file 'instances.json' in the +current working directory, and will write to 'instances.md' in that same directory. WARNING: This script will overwrite the output file if it +exists. + +positional arguments: + INPUT_FILE location of instances JSON + +optional arguments: + -h, --help show this help message and exit + -o OUTPUT_FILE, --output OUTPUT_FILE + where to write the markdown table; if a file exists at this path, it will be overwritten +``` + +### Prerequisites + +`generate-instances-markdown.py` requires **python3** of at least 3.5. + # License -The script `generate-instances-json.sh` and the schema file `instances-schema.json` are licensed under [the GNU General Public License v3.0](https://www.gnu.org/licenses/gpl-3.0.en.html). `instances.json` and `instances.txt` are released to the public domain. +The script `generate-instances-json.sh` and the schema file `instances-schema.json` are licensed under [the GNU General Public License v3.0](https://www.gnu.org/licenses/gpl-3.0.en.html). Almost all of `generate-instances-markdown.py` is licensed under GPL v3.0, with the exception of portions of MIT-licensed code adapted from [Django Countries](https://github.com/SmileyChris/django-countries) which generates a [regional indicator symbol](https://en.wikipedia.org/wiki/Regional_indicator_symbol) for a given [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country code; view the source for `generate-instances-markdown.py` for the applicable code along with a copy of the MIT License as it appeared in the Django Countries license at the time the code was adapted. + +`instances.json`, `instances.md`, and `instances.txt` are released to the public domain. diff --git a/generate-instances-markdown.py b/generate-instances-markdown.py new file mode 100755 index 0000000..5ee6604 --- /dev/null +++ b/generate-instances-markdown.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 + +""" +This script generates a markdown table containing the list of Libreddit +instances from an instances JSON file. It is assumed that the input instances +JSON file follows the schema as instances-schema.json. + +This script requires python3 of at least version 3.5. + +Almost all of this script is licensed under the GNU General Public License, +version 3. + +Portions of this script, specifically the function `flag()`, are adapted from +Django Countries (https://github.com/SmileyChris/django-countries), licensed +under the MIT License. Pursuant to the copyright notice requirement of that +license, the full contents of the license agreement as of Nov 5, 2022, are +reproduced below. + + Copyright (c) 2010 Chris Beaven and contributors + + Permission is hereby granted, free of charge, to any person + obtaining a copy of this software and associated documentation + files (the "Software"), to deal in the Software without + restriction, including without limitation the rights to use, + copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the + Software is furnished to do so, subject to the following + conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + OTHER DEALINGS IN THE SOFTWARE. +""" + +import argparse +import json +import os +import sys + +# This script requires at least Python 3.5. +if sys.version_info[1] < 5: + raise SystemError("Your python3 ({}.{}.{}) is too old. You need at least 3.5.".format(sys.version_info[0], sys.version_info[1], sys.version_info[2])) + +def flag(code: str) -> str: + """ + Generate a regional indicator symbol from an ISO 3166-1 alpha-2 country + code. + + This code is adapted from Django Countries: + https://github.com/SmileyChris/django-countries/blob/732030e5c912875927fcc012e0bb2e392ae10a0b/django_countries/fields.py#L143 + which at the time of adaptation was licensed under MIT: + https://github.com/SmileyChris/django-countries/blob/732030e5c912875927fcc012e0bb2e392ae10a0b/LICENSE + which permits modification and distribution of code. + """ + + OFFSET = 127397 + + if not code: + return "" + + points = [ord(x) + OFFSET for x in code.upper()] + return chr(points[0]) + chr(points[1]) + +def main(args: list) -> int: + """ + Main program function. Does everything the script is supposed to do. + """ + + # Set up options and parse arguments. + parser = argparse.ArgumentParser(description=""" +Generate a markdown table of the Libreddit instances in the instances.JSON +file. By default, this will read the file 'instances.json' in the current +working directory, and will write to 'instances.md' in that same directory. +WARNING: This script will overwrite the output file if it exists. +""") + parser.add_argument("INPUT_FILE", default="instances.json", nargs="?", + help="location of instances JSON") + parser.add_argument("-o", "--output", dest="OUTPUT_FILE", + default="instances.md", help="where to write the markdown table; \ + if a file exists at this path, it will be overwritten") + + parsed_args = parser.parse_args(args[1:]) + + try: + with open(parsed_args.INPUT_FILE) as f: + instances = json.load(f) + except Exception as e: + sys.stderr.write("Error opening '{}' for reading:\n".format(parsed_args.INPUT_FILE)) + sys.stderr.write("\t" + e.__str__() + "\n") + return 1 + + if parsed_args.OUTPUT_FILE == "-": + out = sys.stdout + else: + try: + mode="x" + if os.path.exists(parsed_args.OUTPUT_FILE): + mode="w" + out = open(parsed_args.OUTPUT_FILE, mode) + except Exception as e: + sys.stderr.write("Error opening '{}' for writing:\n".format(parsed_args.OUTPUT_FILE)) + sys.stderr.write("\t" + e.__str__() + "\n") + return 1 + + table_preamble = "|URL|Network|Version|Location|Behind Cloudflare?|Comment|\n|-|-|-|-|-|-|\n" + table_rows = [] + for instance in instances["instances"]: + url = "" + network = "" + version = "" + country = "(n/a)" + cloudflare = False + description = "" + + if "url" in instance: + url = instance["url"] + network = "WWW" + elif "onion" in instance: + url = instance["onion"] + network = "Tor" + elif "i2p" in instance: + url = instance["i2p"] + network = "I2P" + else: + # Couldn't determine network, so skip instance. + continue + + # Version is a required parameter. If this is not in the JSON, skip + # this instance. + if "version" not in instance: + sys.stderr.write("Skipping '{}': no version recorded".format(url)) + continue + else: + version = instance["version"] + + if "country" in instance: + country = instance["country"] + + if "cloudflare" in instance and instance["cloudflare"]: + cloudflare = True + + if "description" in instance: + description = instance["description"] + + location = "" + try: + fl = flag(country) + location = fl + " " + country + except Exception: + location = country + + table_rows.append("|{0}|{1}|{2}|{3}|{4}|{5}|\n".format( + url, + network, + version, + location, + "\u2705" if cloudflare else "", + description + )) + + out.write(table_preamble) + for row in table_rows: + out.write(row) + out.close() + + return 0 + +if __name__ == "__main__": + rc = main(sys.argv) + sys.exit(rc) diff --git a/instances.json b/instances.json index 8847f40..b6fcc1b 100644 --- a/instances.json +++ b/instances.json @@ -234,11 +234,6 @@ "country": "FI", "version": "v0.22.9" }, - { - "url": "https://lr.slipfox.xyz", - "country": "US", - "version": "v0.22.9" - }, { "url": "https://libreddit.garudalinux.org", "country": "FI", diff --git a/instances.md b/instances.md new file mode 100644 index 0000000..30e42c4 --- /dev/null +++ b/instances.md @@ -0,0 +1,54 @@ +|URL|Network|Version|Location|Behind Cloudflare?|Comment| +|-|-|-|-|-|-| +|https://libreddit.spike.codes|WWW|v0.23.1|🇺🇸 US||official instance| +|https://libreddit.kavin.rocks|WWW|v0.23.2|🇮🇳 IN||| +|https://reddit.invak.id|WWW|v0.23.2|🇧🇬 BG||| +|https://reddit.simo.sh|WWW|v0.23.2|🇧🇬 BG||| +|https://lr.riverside.rocks|WWW|v0.15.2|🇺🇸 US||| +|https://libreddit.strongthany.cc|WWW|v0.22.9|🇺🇸 US||| +|https://libreddit.privacy.com.de|WWW|v0.22.9|🇩🇪 DE||| +|https://libreddit.domain.glass|WWW|v0.10.5|🇺🇸 US|✅|| +|https://r.nf|WWW|v0.23.1|🇩🇪 DE|✅|| +|https://reddit.stuehieyr.com|WWW|v0.22.6|🇩🇪 DE||| +|https://lr.mint.lgbt|WWW|v0.22.9|🇨🇦 CA||| +|https://libreddit.drivet.xyz|WWW|v0.23.2|🇵🇱 PL||| +|https://libreddit.de|WWW|v0.22.9|🇩🇪 DE||| +|https://libreddit.pussthecat.org|WWW|v0.23.2|🇩🇪 DE||| +|https://libreddit.northboot.xyz|WWW|v0.23.2|🇩🇪 DE||| +|https://libreddit.hu|WWW|v0.20.2|🇫🇮 FI|✅|| +|https://libreddit.totaldarkness.net|WWW|v0.23.1|🇨🇦 CA||| +|https://libreddit.esmailelbob.xyz|WWW|v0.23.1|🇨🇦 CA||| +|https://lr.vern.cc|WWW|v0.1.0|🇨🇦 CA||| +|https://libreddit.nl|WWW|v0.23.2|🇳🇱 NL||| +|https://reddi.tk|WWW|v0.22.9|🇺🇸 US|✅|| +|https://r.walkx.org|WWW|v0.22.8|🇳🇱 NL|✅|| +|https://libreddit.kylrth.com|WWW|v0.22.9|🇨🇦 CA||| +|https://libreddit.tiekoetter.com|WWW|v0.23.1|🇩🇪 DE||| +|https://reddit.rtrace.io|WWW|v0.23.2|🇩🇪 DE||| +|https://libreddit.privacydev.net|WWW|v0.22.9|🇺🇸 US||| +|https://r.ahwx.org|WWW|v0.22.9|🇳🇱 NL|✅|| +|https://bob.fr.to|WWW|v0.23.1|🇺🇸 US||| +|https://libreddit.dcs0.hu|WWW|v0.22.8|🇭🇺 HU||| +|https://reddit.dr460nf1r3.org|WWW|v0.23.2|🇩🇪 DE|✅|| +|https://rd.jae.su|WWW|v0.23.2|🇫🇮 FI||| +|https://libreddit.foss.wtf|WWW|v0.23.2|🇩🇪 DE||| +|https://libreddit.encrypted-data.xyz|WWW|v0.22.9|🇫🇷 FR|✅|| +|https://libreddit.eu.org|WWW|v0.22.9|🇮🇪 IE|✅|| +|https://l.opnxng.com|WWW|v0.23.2|🇸🇬 SG||| +|https://libreddit.cachyos.org|WWW|v0.22.9|🇩🇪 DE|✅|| +|https://rd.funami.tech|WWW|v0.22.8|🇰🇷 KR||| +|https://libreddit.projectsegfau.lt|WWW|v0.22.9|🇱🇺 LU||| +|https://www.futureddit.gq|WWW|v0.23.1|🇹🇷 TR|✅|| +|https://lr.slipfox.xyz|WWW|v0.22.9|🇺🇸 US||| +|https://libreddit.oxymagnesium.com|WWW|v0.22.9|🇺🇸 US||| +|https://reddit.utsav2.dev|WWW|v0.22.9|🇺🇸 US||| +|https://libreddit.freedit.eu|WWW|v0.23.1|🇺🇸 US|✅|| +|https://libreddit.mha.fi|WWW|v0.22.9|🇫🇮 FI||| +|https://libreddit.garudalinux.org|WWW|v0.23.2|🇫🇮 FI||| +|https://libreddit.mask.sh|WWW|v0.23.2|🇺🇸 US||| +|https://www.futureddit.gq|WWW|v0.23.1|🇮🇹 IT|✅|| +|https://lr.4201337.xyz|WWW|v0.22.9|🇫🇷 FR||| +|http://ojwp2gtj7dq7scd7gnbac6wp53tklgsicteabrnx2pr7zai64wriiaad.onion|Tor|v0.22.9|🇺🇸 US||| +|http://ecue64ybzvn6vjzl37kcsnwt4ycmbsyf74nbttyg7rkc3t3qwnj7mcyd.onion|Tor|v0.23.2|🇩🇪 DE||| +|http://libreddit.esmail5pdn24shtvieloeedh7ehz3nrwcdivnfhfcedl7gf4kwddhkqd.onion|Tor|v0.23.1|🇨🇦 CA||| +|http://ojwp2gtj7dq7scd7gnbac6wp53tklgsicteabrnx2pr7zai64wriiaad.onion|Tor|v0.22.9|🇺🇸 US||| diff --git a/instances.txt b/instances.txt index 64240b5..f89a97e 100644 --- a/instances.txt +++ b/instances.txt @@ -43,7 +43,6 @@ https://libreddit.oxymagnesium.com,US,false,"" https://reddit.utsav2.dev,US,false,"" https://libreddit.freedit.eu,US,true,"" https://libreddit.mha.fi,FI,false,"" -https://lr.slipfox.xyz,US,false,"" https://libreddit.garudalinux.org,FI,false,"" https://libreddit.mask.sh,US,false,"" https://www.futureddit.gq,IT,true,""