redlib-org/redlib-instances
1
0
Fork 0
mirror of https://github.com/redlib-org/redlib-instances.git synced 2024-11-13 18:06:15 -05:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Add instances.md and script for generating said.

Browse Source
This commit is contained in:
Daniel Valentine 2022-11-05 20:43:58 -06:00
parent aff6919a95
commit 5cb47d4510
5 changed files with 264 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
README.md
View File

@ -9,6 +9,7 @@ This repo consists of four files:
1. `instances.json`: This is the list of Libreddit instances.
1. `instances-schema.json`: JSON Schema governing `instances.json`.
1. `instances.txt`: This is a CSV of libreddit instances. While this is also machine-readable, it is recommended to use `instances.json` instead. `instances.txt` is meant for contributors to add and remove instances, and `generate-instances-json.sh` will validate those instances and generate `instances.json`.
1. `instances.md`: This is table in Markdown format of the Libreddit instances in `instances.json`. It is generated by the script `generate-instances-markdown.py`.
1. `generate-instances-json.sh`: This script takes in a CSV file as input, typically `instances.txt`, and outputs a JSON object with a list of Libreddit instances. This is the script that generates `instances.json`.
## Adding or removing an instance
@ -17,12 +18,13 @@ To generate `instances.json`, perform the following:
1. Modify `instances.txt` to add or remove instances. See [Expected CSV format](#Expected CSV format) for the expected format of each CSV row.
1. Run `generate-instances-json.sh -i ./instances.txt -o ./instances.json` to generate `instances.json`. The existing `instances.json` will be replaced.
1. Run `generate-instances-markdown.py --output=./instances.md ./instances.json` to generate `instances.md`. The existing `instances.md` file will be replaced.
Pull requests to add or remove instances are always welcome.
## `generate-instances-json.sh`
`generate-instances-json.sh` is the script that produces a JSON of [Libreddit](https://github.com/spikecodes/libreddit) instances, given a CSV input of Libreddit instances.
`generate-instances-json.sh` is the script that produces a JSON of [Libreddit](https://github.com/libreddit/libreddit) instances, given a CSV input of Libreddit instances.
Unless `-i` and `-o` are specified (see [Usage](#Usage) below), the input and output are assumed the stdin and stdout streams respectively.
@ -131,6 +133,34 @@ If all instances could be processed, the script exits with an exit code of 0. If
This script will attempt to connect to instances that are onion sites. To make sure it can do this, it will see if Tor is running and if torsocks is installed. If neither condition is met, the script will not attempt to connect to Libreddit onion sites and will skip them. The exit code will still be 0, assuming that the WWW Libreddit sites were processed without error.
## `generate-instances-markdown.py`
`generate-instances-markdown.py` will generate a table in Markdown format of the instances. This requires the JSON file that is generated by `generate-instances-json.sh`.
### Usage
```
usage: generate-instances-markdown.py [-h] [-o OUTPUT_FILE] [INPUT_FILE]
Generate a markdown table of the Libreddit instances in the instances.JSON file. By default, this will read the file 'instances.json' in the
current working directory, and will write to 'instances.md' in that same directory. WARNING: This script will overwrite the output file if it
exists.
positional arguments:
INPUT_FILE location of instances JSON
optional arguments:
-h, --help show this help message and exit
-o OUTPUT_FILE, --output OUTPUT_FILE
where to write the markdown table; if a file exists at this path, it will be overwritten
```
### Prerequisites
`generate-instances-markdown.py` requires **python3** of at least 3.5.
# License
The script `generate-instances-json.sh` and the schema file `instances-schema.json` are licensed under [the GNU General Public License v3.0](https://www.gnu.org/licenses/gpl-3.0.en.html). `instances.json` and `instances.txt` are released to the public domain.
The script `generate-instances-json.sh` and the schema file `instances-schema.json` are licensed under [the GNU General Public License v3.0](https://www.gnu.org/licenses/gpl-3.0.en.html). Almost all of `generate-instances-markdown.py` is licensed under GPL v3.0, with the exception of portions of MIT-licensed code adapted from [Django Countries](https://github.com/SmileyChris/django-countries) which generates a [regional indicator symbol](https://en.wikipedia.org/wiki/Regional_indicator_symbol) for a given [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) country code; view the source for `generate-instances-markdown.py` for the applicable code along with a copy of the MIT License as it appeared in the Django Countries license at the time the code was adapted.
`instances.json`, `instances.md`, and `instances.txt` are released to the public domain.

178
generate-instances-markdown.py Executable file
View File

@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""
This script generates a markdown table containing the list of Libreddit
instances from an instances JSON file. It is assumed that the input instances
JSON file follows the schema as instances-schema.json.
This script requires python3 of at least version 3.5.
Almost all of this script is licensed under the GNU General Public License,
version 3.
Portions of this script, specifically the function `flag()`, are adapted from
Django Countries (https://github.com/SmileyChris/django-countries), licensed
under the MIT License. Pursuant to the copyright notice requirement of that
license, the full contents of the license agreement as of Nov 5, 2022, are
reproduced below.
Copyright (c) 2010 Chris Beaven and contributors
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
"""
import argparse
import json
import os
import sys
# This script requires at least Python 3.5.
if sys.version_info[1] < 5:
raise SystemError("Your python3 ({}.{}.{}) is too old. You need at least 3.5.".format(sys.version_info[0], sys.version_info[1], sys.version_info[2]))
def flag(code: str) -> str:
"""
Generate a regional indicator symbol from an ISO 3166-1 alpha-2 country
code.
This code is adapted from Django Countries:
https://github.com/SmileyChris/django-countries/blob/732030e5c912875927fcc012e0bb2e392ae10a0b/django_countries/fields.py#L143
which at the time of adaptation was licensed under MIT:
https://github.com/SmileyChris/django-countries/blob/732030e5c912875927fcc012e0bb2e392ae10a0b/LICENSE
which permits modification and distribution of code.
"""
OFFSET = 127397
if not code:
return ""
points = [ord(x) + OFFSET for x in code.upper()]
return chr(points[0]) + chr(points[1])
def main(args: list) -> int:
"""
Main program function. Does everything the script is supposed to do.
"""
# Set up options and parse arguments.
parser = argparse.ArgumentParser(description="""
Generate a markdown table of the Libreddit instances in the instances.JSON
file. By default, this will read the file 'instances.json' in the current
working directory, and will write to 'instances.md' in that same directory.
WARNING: This script will overwrite the output file if it exists.
""")
parser.add_argument("INPUT_FILE", default="instances.json", nargs="?",
help="location of instances JSON")
parser.add_argument("-o", "--output", dest="OUTPUT_FILE",
default="instances.md", help="where to write the markdown table; \
if a file exists at this path, it will be overwritten")
parsed_args = parser.parse_args(args[1:])
try:
with open(parsed_args.INPUT_FILE) as f:
instances = json.load(f)
except Exception as e:
sys.stderr.write("Error opening '{}' for reading:\n".format(parsed_args.INPUT_FILE))
sys.stderr.write("\t" + e.__str__() + "\n")
return 1
if parsed_args.OUTPUT_FILE == "-":
out = sys.stdout
else:
try:
mode="x"
if os.path.exists(parsed_args.OUTPUT_FILE):
mode="w"
out = open(parsed_args.OUTPUT_FILE, mode)
except Exception as e:
sys.stderr.write("Error opening '{}' for writing:\n".format(parsed_args.OUTPUT_FILE))
sys.stderr.write("\t" + e.__str__() + "\n")
return 1
table_preamble = "|URL|Network|Version|Location|Behind Cloudflare?|Comment|\n|-|-|-|-|-|-|\n"
table_rows = []
for instance in instances["instances"]:
url = ""
network = ""
version = ""
country = "(n/a)"
cloudflare = False
description = ""
if "url" in instance:
url = instance["url"]
network = "WWW"
elif "onion" in instance:
url = instance["onion"]
network = "Tor"
elif "i2p" in instance:
url = instance["i2p"]
network = "I2P"
else:
# Couldn't determine network, so skip instance.
continue
# Version is a required parameter. If this is not in the JSON, skip
# this instance.
if "version" not in instance:
sys.stderr.write("Skipping '{}': no version recorded".format(url))
continue
else:
version = instance["version"]
if "country" in instance:
country = instance["country"]
if "cloudflare" in instance and instance["cloudflare"]:
cloudflare = True
if "description" in instance:
description = instance["description"]
location = ""
try:
fl = flag(country)
location = fl + " " + country
except Exception:
location = country
table_rows.append("|{0}|{1}|{2}|{3}|{4}|{5}|\n".format(
url,
network,
version,
location,
"\u2705" if cloudflare else "",
description
))
out.write(table_preamble)
for row in table_rows:
out.write(row)
out.close()
return 0
if __name__ == "__main__":
rc = main(sys.argv)
sys.exit(rc)

5
instances.json
View File

@ -234,11 +234,6 @@
"country": "FI",
"version": "v0.22.9"
},
{
"url": "https://lr.slipfox.xyz",
"country": "US",
"version": "v0.22.9"
},
{
"url": "https://libreddit.garudalinux.org",
"country": "FI",

54
instances.md Normal file
View File

@ -0,0 +1,54 @@
|URL|Network|Version|Location|Behind Cloudflare?|Comment|
|-|-|-|-|-|-|
|https://libreddit.spike.codes|WWW|v0.23.1|🇺🇸 US||official instance|
|https://libreddit.kavin.rocks|WWW|v0.23.2|🇮🇳 IN|||
|https://reddit.invak.id|WWW|v0.23.2|🇧🇬 BG|||
|https://reddit.simo.sh|WWW|v0.23.2|🇧🇬 BG|||
|https://lr.riverside.rocks|WWW|v0.15.2|🇺🇸 US|||
|https://libreddit.strongthany.cc|WWW|v0.22.9|🇺🇸 US|||
|https://libreddit.privacy.com.de|WWW|v0.22.9|🇩🇪 DE|||
|https://libreddit.domain.glass|WWW|v0.10.5|🇺🇸 US|✅||
|https://r.nf|WWW|v0.23.1|🇩🇪 DE|✅||
|https://reddit.stuehieyr.com|WWW|v0.22.6|🇩🇪 DE|||
|https://lr.mint.lgbt|WWW|v0.22.9|🇨🇦 CA|||
|https://libreddit.drivet.xyz|WWW|v0.23.2|🇵🇱 PL|||
|https://libreddit.de|WWW|v0.22.9|🇩🇪 DE|||
|https://libreddit.pussthecat.org|WWW|v0.23.2|🇩🇪 DE|||
|https://libreddit.northboot.xyz|WWW|v0.23.2|🇩🇪 DE|||
|https://libreddit.hu|WWW|v0.20.2|🇫🇮 FI|✅||
|https://libreddit.totaldarkness.net|WWW|v0.23.1|🇨🇦 CA|||
|https://libreddit.esmailelbob.xyz|WWW|v0.23.1|🇨🇦 CA|||
|https://lr.vern.cc|WWW|v0.1.0|🇨🇦 CA|||
|https://libreddit.nl|WWW|v0.23.2|🇳🇱 NL|||
|https://reddi.tk|WWW|v0.22.9|🇺🇸 US|✅||
|https://r.walkx.org|WWW|v0.22.8|🇳🇱 NL|✅||
|https://libreddit.kylrth.com|WWW|v0.22.9|🇨🇦 CA|||
|https://libreddit.tiekoetter.com|WWW|v0.23.1|🇩🇪 DE|||
|https://reddit.rtrace.io|WWW|v0.23.2|🇩🇪 DE|||
|https://libreddit.privacydev.net|WWW|v0.22.9|🇺🇸 US|||
|https://r.ahwx.org|WWW|v0.22.9|🇳🇱 NL|✅||
|https://bob.fr.to|WWW|v0.23.1|🇺🇸 US|||
|https://libreddit.dcs0.hu|WWW|v0.22.8|🇭🇺 HU|||
|https://reddit.dr460nf1r3.org|WWW|v0.23.2|🇩🇪 DE|✅||
|https://rd.jae.su|WWW|v0.23.2|🇫🇮 FI|||
|https://libreddit.foss.wtf|WWW|v0.23.2|🇩🇪 DE|||
|https://libreddit.encrypted-data.xyz|WWW|v0.22.9|🇫🇷 FR|✅||
|https://libreddit.eu.org|WWW|v0.22.9|🇮🇪 IE|✅||
|https://l.opnxng.com|WWW|v0.23.2|🇸🇬 SG|||
|https://libreddit.cachyos.org|WWW|v0.22.9|🇩🇪 DE|✅||
|https://rd.funami.tech|WWW|v0.22.8|🇰🇷 KR|||
|https://libreddit.projectsegfau.lt|WWW|v0.22.9|🇱🇺 LU|||
|https://www.futureddit.gq|WWW|v0.23.1|🇹🇷 TR|✅||
|https://lr.slipfox.xyz|WWW|v0.22.9|🇺🇸 US|||
|https://libreddit.oxymagnesium.com|WWW|v0.22.9|🇺🇸 US|||
|https://reddit.utsav2.dev|WWW|v0.22.9|🇺🇸 US|||
|https://libreddit.freedit.eu|WWW|v0.23.1|🇺🇸 US|✅||
|https://libreddit.mha.fi|WWW|v0.22.9|🇫🇮 FI|||
|https://libreddit.garudalinux.org|WWW|v0.23.2|🇫🇮 FI|||
|https://libreddit.mask.sh|WWW|v0.23.2|🇺🇸 US|||
|https://www.futureddit.gq|WWW|v0.23.1|🇮🇹 IT|✅||
|https://lr.4201337.xyz|WWW|v0.22.9|🇫🇷 FR|||
|http://ojwp2gtj7dq7scd7gnbac6wp53tklgsicteabrnx2pr7zai64wriiaad.onion|Tor|v0.22.9|🇺🇸 US|||
|http://ecue64ybzvn6vjzl37kcsnwt4ycmbsyf74nbttyg7rkc3t3qwnj7mcyd.onion|Tor|v0.23.2|🇩🇪 DE|||
|http://libreddit.esmail5pdn24shtvieloeedh7ehz3nrwcdivnfhfcedl7gf4kwddhkqd.onion|Tor|v0.23.1|🇨🇦 CA|||
|http://ojwp2gtj7dq7scd7gnbac6wp53tklgsicteabrnx2pr7zai64wriiaad.onion|Tor|v0.22.9|🇺🇸 US|||

1
instances.txt
View File

@ -43,7 +43,6 @@ https://libreddit.oxymagnesium.com,US,false,""
https://reddit.utsav2.dev,US,false,""
https://libreddit.freedit.eu,US,true,""
https://libreddit.mha.fi,FI,false,""
https://lr.slipfox.xyz,US,false,""
https://libreddit.garudalinux.org,FI,false,""
https://libreddit.mask.sh,US,false,""
https://www.futureddit.gq,IT,true,""