redlib-org/redlib-instances
1
0
Fork 0
mirror of https://github.com/redlib-org/redlib-instances.git synced 2024-11-13 18:06:15 -05:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Update generate-instances-json.sh - replace torsocks with curl socks proxy

Browse Source
This commit is contained in:
Matthew Esposito 2024-01-17 10:53:26 -05:00
parent bb91b5ca61
commit 86bb04e1f2
No known key found for this signature in database
2 changed files with 16 additions and 15 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
README.md
View File

@ -58,9 +58,9 @@ DESCRIPTION
This proxy typically listens at 127.0.0.1:4444. This proxy typically listens at 127.0.0.1:4444.
* This script will attempt to connect to instances in the CSV that are on * This script will attempt to connect to instances in the CSV that are on
Tor, provided that it can (it will check to see if Tor is running and the Tor, provided that it can (it will check to see if Tor is running).
availability of the torsocks program). If you want to disable connections If you want to disable connections to these onion sites, provide the
to these onion sites, provide the -T option. -T option.
* This script will return a non-zero status code when at least one instance * This script will return a non-zero status code when at least one instance
could not be reached. If you want this script always to return 0 even could not be reached. If you want this script always to return 0 even
@ -124,7 +124,8 @@ ENVIRONMENT
`generate-instances-json.sh` requires **curl** in order to make HTTP(S) requests and **jq** to process and format JSON. `generate-instances-json.sh` requires **curl** in order to make HTTP(S) requests and **jq** to process and format JSON.
**tor** and **torsocks** are required for processing onion sites, but the script will skip instances on Tor if neither tor is running nor torsocks is available. An option exists to import onion sites from an existing JSON file should you wish not to use tor. **tor** is required for processing onion sites, but the script will skip instances on Tor if tor is not running.
An option exists to import onion sites from an existing JSON file should you wish not to use tor.
### Expected CSV format ### Expected CSV format
@ -170,7 +171,7 @@ This script will attempt to connect to instances that are onion or I2P sites.
##### Tor ##### Tor
To make sure it can connect to onion sites, the script will see if Tor is running and if torsocks is installed. If neither condition is met, the script will not attempt to connect to Redlib onion sites and will skip them. The exit code will still be 0, assuming that the WWW Redlib sites were processed without error. To make sure it can connect to onion sites, the script will see if Tor is running. If it is not, the script will not attempt to connect to Redlib onion sites and will skip them. The exit code will still be 0, assuming that the WWW Redlib sites were processed without error.
##### I2P ##### I2P

20
generate-instances-json.sh
View File

@ -61,10 +61,10 @@ check_program ()
# can_tor # can_tor
# #
# Returns true if tor is running and torsocks is installed. # Returns true if tor is running
can_tor () can_tor ()
{ {
check_tor && check_program torsocks check_tor
} }
# can_i2p # can_i2p
@ -273,7 +273,7 @@ canonicalize_url ()
# #
# Makes an HTTP(S) GET equest to the provided URL with curl. The response is # Makes an HTTP(S) GET equest to the provided URL with curl. The response is
# written to standard out. get will determine if the URL is an onion site, and, # written to standard out. get will determine if the URL is an onion site, and,
# if so, it wrap the curl call with torsocks. If the URL is a I2P site, and # if so, it wrap the curl call with socks proxy. If the URL is a I2P site, and
# I2P_HTTP_PROXY is non-empty, tell curl to use that as the proxy. # I2P_HTTP_PROXY is non-empty, tell curl to use that as the proxy.
# #
# The return value is the curl return value, or: # The return value is the curl return value, or:
@ -345,8 +345,8 @@ get ()
# - Increase curl max-time to 60 seconds. # - Increase curl max-time to 60 seconds.
if [[ "${zone,,}" == "onion" ]] if [[ "${zone,,}" == "onion" ]]
then then
# Don't bother if tor isn't running or we don't have torsocks. But if # Don't bother if tor isn't running. But if both are available,
# both are available, make sure we warp curl with torsocks. # make sure we warp curl with socks.
if [[ "${no_tor}" == "y" ]] if [[ "${no_tor}" == "y" ]]
then then
return 104 return 104
@ -358,7 +358,7 @@ get ()
fi fi
timeout=60 timeout=60
curl_cmd=(torsocks curl) curl_cmd=(curl --proxy socks5h://localhost:9050)
elif [[ "${zone,,}" == "i2p" ]] elif [[ "${zone,,}" == "i2p" ]]
then then
if [[ "${no_i2p}" == "y" ]] if [[ "${no_i2p}" == "y" ]]
@ -552,9 +552,9 @@ DESCRIPTION
This proxy typically listens at 127.0.0.1:4444. This proxy typically listens at 127.0.0.1:4444.
* This script will attempt to connect to instances in the CSV that are on * This script will attempt to connect to instances in the CSV that are on
Tor, provided that it can (it will check to see if Tor is running and the Tor, provided that it can (it will check to see if Tor is running).
availability of the torsocks program). If you want to disable connections If you want to disable connections to these onion sites, provide the
to these onion sites, provide the -T option. -T option.
* This script will return a non-zero status code when at least one instance * This script will return a non-zero status code when at least one instance
could not be reached. If you want this script always to return 0 even could not be reached. If you want this script always to return 0 even
@ -745,7 +745,7 @@ main ()
then then
if [[ "${do_tor}" == "y" ]] if [[ "${do_tor}" == "y" ]]
then then
echo >&2 "WARNING: Either the tor service is not running or torsocks is not available. Either way, onion sites will not be processed." echo >&2 "WARNING: The tor service is not running. Onion sites will not be processed."
fi fi
do_tor="n" do_tor="n"
get_opts+=("-T") get_opts+=("-T")