torproject/tor
1
0
Fork 0
mirror of https://gitlab.torproject.org/tpo/core/tor.git synced 2024-12-06 19:41:15 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Remove script to clean up GeoLite country databases.

Browse Source
This commit is contained in:
Karsten Loesing
2014-02-08 11:52:33 +01:00
parent a0577aacb4
commit becc3d5b4e
3 changed files with 0 additions and 375 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/config/README.geoip
-90
View File
@@ -1,90 +0,0 @@
README.geoip -- information on the IP-to-country-code file shipped with tor
===========================================================================
The IP-to-country-code file in src/config/geoip is based on MaxMind's
GeoLite Country database with the following modifications:
- Those "A1" ("Anonymous Proxy") entries lying inbetween two entries with
the same country code are automatically changed to that country code.
These changes can be overriden by specifying a different country code
in src/config/geoip-manual.
- Other "A1" entries are replaced with country codes specified in
src/config/geoip-manual, or are left as is if there is no corresponding
entry in that file. Even non-"A1" entries can be modified by adding a
replacement entry to src/config/geoip-manual. Handle with care.
1. Updating the geoip file from a MaxMind database file
-------------------------------------------------------
Download the most recent MaxMind GeoLite Country database:
http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
Run `python deanonymind.py` in the local directory. Review the output to
learn about applied automatic/manual changes and watch out for any
warnings.
Possibly edit geoip-manual to make more/fewer/different manual changes and
re-run `python deanonymind.py`.
When done, prepend the new geoip file with a comment like this:
# Last updated based on $DATE Maxmind GeoLite Country
# See README.geoip for details on the conversion.
2. Verifying automatic and manual changes using diff
----------------------------------------------------
To unzip the original MaxMind file and look at the automatic changes, run:
unzip GeoIPCountryCSV.zip
diff -U1 GeoIPCountryWhois.csv AutomaticGeoIPCountryWhois.csv
To look at subsequent manual changes, run:
diff -U1 AutomaticGeoIPCountryWhois.csv ManualGeoIPCountryWhois.csv
To manually generate the geoip file and compare it to the automatically
created one, run:
cut -d, -f3-5 < ManualGeoIPCountryWhois.csv | sed 's/"//g' > mygeoip
diff -U1 geoip mygeoip
3. Verifying automatic and manual changes using blockfinder
-----------------------------------------------------------
Blockfinder is a powerful tool to handle multiple IP-to-country data
sources. Blockfinder has a function to specify a country code and compare
conflicting country code assignments in different data sources.
We can use blockfinder to compare A1 entries in the original MaxMind file
with the same or overlapping blocks in the file generated above and in the
RIR delegation files:
git clone https://github.com/ioerror/blockfinder
cd blockfinder/
python blockfinder -i
python blockfinder -r ../GeoIPCountryWhois.csv
python blockfinder -r ../ManualGeoIPCountryWhois.csv
python blockfinder -p A1 > A1-comparison.txt
The output marks conflicts between assignments using either '*' in case of
two different opinions or '#' for three or more different opinions about
the country code for a given block.
The '*' conflicts are most likely harmless, because there will always be
at least two opinions with the original MaxMind file saying A1 and the
other two sources saying something more meaningful.
However, watch out for '#' conflicts. In these cases, the original
MaxMind file ("A1"), the updated MaxMind file (hopefully the correct
country code), and the RIR delegation files (some other country code) all
disagree.
There are perfectly valid cases where the updated MaxMind file and the RIR
delegation files don't agree. But each of those cases must be verified
manually.
src/config/deanonymind.py
-205
View File
@@ -1,205 +0,0 @@
#!/usr/bin/env python
import optparse
import os
import sys
import zipfile
"""
Take a MaxMind GeoLite Country database as input and replace A1 entries
with the country code and name of the preceding entry iff the preceding
(subsequent) entry ends (starts) directly before (after) the A1 entry and
both preceding and subsequent entries contain the same country code.
Then apply manual changes, either replacing A1 entries that could not be
replaced automatically or overriding previously made automatic changes.
"""
def main():
options = parse_options()
assignments = read_file(options.in_maxmind)
assignments = apply_automatic_changes(assignments)
write_file(options.out_automatic, assignments)
manual_assignments = read_file(options.in_manual, must_exist=False)
assignments = apply_manual_changes(assignments, manual_assignments)
write_file(options.out_manual, assignments)
write_file(options.out_geoip, assignments, long_format=False)
def parse_options():
parser = optparse.OptionParser()
parser.add_option('-i', action='store', dest='in_maxmind',
default='GeoIPCountryCSV.zip', metavar='FILE',
help='use the specified MaxMind GeoLite Country .zip or .csv '
'file as input [default: %default]')
parser.add_option('-g', action='store', dest='in_manual',
default='geoip-manual', metavar='FILE',
help='use the specified .csv file for manual changes or to '
'override automatic changes [default: %default]')
parser.add_option('-a', action='store', dest='out_automatic',
default="AutomaticGeoIPCountryWhois.csv", metavar='FILE',
help='write full input file plus automatic changes to the '
'specified .csv file [default: %default]')
parser.add_option('-m', action='store', dest='out_manual',
default='ManualGeoIPCountryWhois.csv', metavar='FILE',
help='write full input file plus automatic and manual '
'changes to the specified .csv file [default: %default]')
parser.add_option('-o', action='store', dest='out_geoip',
default='geoip', metavar='FILE',
help='write full input file plus automatic and manual '
'changes to the specified .csv file that can be shipped '
'with tor [default: %default]')
(options, args) = parser.parse_args()
return options
def read_file(path, must_exist=True):
if not os.path.exists(path):
if must_exist:
print 'File %s does not exist. Exiting.' % (path, )
sys.exit(1)
else:
return
if path.endswith('.zip'):
zip_file = zipfile.ZipFile(path)
csv_content = zip_file.read('GeoIPCountryWhois.csv')
zip_file.close()
else:
csv_file = open(path)
csv_content = csv_file.read()
csv_file.close()
assignments = []
for line in csv_content.split('\n'):
stripped_line = line.strip()
if len(stripped_line) > 0 and not stripped_line.startswith('#'):
assignments.append(stripped_line)
return assignments
def apply_automatic_changes(assignments):
print '\nApplying automatic changes...'
result_lines = []
prev_line = None
a1_lines = []
for line in assignments:
if '"A1"' in line:
a1_lines.append(line)
else:
if len(a1_lines) > 0:
new_a1_lines = process_a1_lines(prev_line, a1_lines, line)
for new_a1_line in new_a1_lines:
result_lines.append(new_a1_line)
a1_lines = []
result_lines.append(line)
prev_line = line
if len(a1_lines) > 0:
new_a1_lines = process_a1_lines(prev_line, a1_lines, None)
for new_a1_line in new_a1_lines:
result_lines.append(new_a1_line)
return result_lines
def process_a1_lines(prev_line, a1_lines, next_line):
if not prev_line or not next_line:
return a1_lines # Can't merge first or last line in file.
if len(a1_lines) > 1:
return a1_lines # Can't merge more than 1 line at once.
a1_line = a1_lines[0].strip()
prev_entry = parse_line(prev_line)
a1_entry = parse_line(a1_line)
next_entry = parse_line(next_line)
touches_prev_entry = int(prev_entry['end_num']) + 1 == \
int(a1_entry['start_num'])
touches_next_entry = int(a1_entry['end_num']) + 1 == \
int(next_entry['start_num'])
same_country_code = prev_entry['country_code'] == \
next_entry['country_code']
if touches_prev_entry and touches_next_entry and same_country_code:
new_line = format_line_with_other_country(a1_entry, prev_entry)
print '-%s\n+%s' % (a1_line, new_line, )
return [new_line]
else:
return a1_lines
def parse_line(line):
if not line:
return None
keys = ['start_str', 'end_str', 'start_num', 'end_num',
'country_code', 'country_name']
stripped_line = line.replace('"', '').strip()
parts = stripped_line.split(',')
entry = dict((k, v) for k, v in zip(keys, parts))
return entry
def format_line_with_other_country(original_entry, other_entry):
return '"%s","%s","%s","%s","%s","%s"' % (original_entry['start_str'],
original_entry['end_str'], original_entry['start_num'],
original_entry['end_num'], other_entry['country_code'],
other_entry['country_name'], )
def apply_manual_changes(assignments, manual_assignments):
if not manual_assignments:
return assignments
print '\nApplying manual changes...'
manual_dict = {}
for line in manual_assignments:
start_num = parse_line(line)['start_num']
if start_num in manual_dict:
print ('Warning: duplicate start number in manual '
'assignments:\n %s\n %s\nDiscarding first entry.' %
(manual_dict[start_num], line, ))
manual_dict[start_num] = line
result = []
for line in assignments:
entry = parse_line(line)
start_num = entry['start_num']
if start_num in manual_dict:
manual_line = manual_dict[start_num]
manual_entry = parse_line(manual_line)
if entry['start_str'] == manual_entry['start_str'] and \
entry['end_str'] == manual_entry['end_str'] and \
entry['end_num'] == manual_entry['end_num']:
if len(manual_entry['country_code']) != 2:
print '-%s' % (line, ) # only remove, don't replace
del manual_dict[start_num]
elif entry['country_code'] != \
manual_entry['country_code']:
new_line = format_line_with_other_country(entry,
manual_entry)
print '-%s\n+%s' % (line, new_line, )
result.append(new_line)
del manual_dict[start_num]
else:
print ('Warning: not applying ineffective manual '
'change:\n %s\n %s' % (line, manual_line, ))
result.append(line)
else:
print ('Warning: not applying manual change that is only '
'a partial match:\n %s\n %s' %
(line, manual_line, ))
result.append(line)
elif 'country_code' in entry and \
entry['country_code'] == 'A1':
print ('Warning: no manual replacement for A1 entry:\n %s'
% (line, ))
result.append(line)
else:
result.append(line)
if len(manual_dict) > 0:
print 'Warning: could not apply all manual assignments:'
for line in manual_dict.values():
print ' %s' % (line, )
return result
def write_file(path, assignments, long_format=True):
if long_format:
output_lines = assignments
else:
output_lines = []
for long_line in assignments:
entry = parse_line(long_line)
short_line = "%s,%s,%s" % (entry['start_num'],
entry['end_num'], entry['country_code'], )
output_lines.append(short_line)
out_file = open(path, 'w')
out_file.write('\n'.join(output_lines))
out_file.close()
if __name__ == '__main__':
main()
src/config/geoip-manual
-80
View File
@@ -1,80 +0,0 @@
# This file contains manual overrides of A1 entries (and possibly others)
# in MaxMind's GeoLite Country database. Use deanonymind.py in the same
# directory to process this file when producing a new geoip file. See
# README.geoip in the same directory for details.
# GB, because RIR delegation files say exactly this range
# 46.16.32.0-46.16.39.255 is GB, even though neither previous nor next
# MaxMind range is GB. Both previous and next MaxMind ranges match RIR
# delegation files, too. -KL 2013-03-07
"46.16.32.0","46.16.39.255","772808704","772810751","GB","United Kingdom"
# CH, because previous MaxMind entry 46.19.141.0-46.19.142.255 is CH, and
# RIR delegation files say 46.19.136.0-46.19.143.255 is CH.
# -KL 2012-11-27
"46.19.143.0","46.19.143.255","773033728","773033983","CH","Switzerland"
# GB, because next MaxMind entry 46.166.129.0-46.166.134.255 is GB, and
# RIR delegation files say 46.166.128.0-46.166.191.255 is GB.
# -KL 2012-11-27
"46.166.128.0","46.166.128.255","782663680","782663935","GB","United Kingdom"
# US, because previous MaxMind entry 70.159.21.51-70.232.244.255 is US,
# because next MaxMind entry 70.232.245.58-70.232.245.59 is A2 ("Satellite
# Provider") which is a country information about as useless as A1, and
# because RIR delegation files say 70.224.0.0-70.239.255.255 is US.
# -KL 2012-11-27
"70.232.245.0","70.232.245.57","1189672192","1189672249","US","United States"
# US, because next MaxMind entry 70.232.246.0-70.240.141.255 is US,
# because previous MaxMind entry 70.232.245.58-70.232.245.59 is A2
# ("Satellite Provider") which is a country information about as useless
# as A1, and because RIR delegation files say 70.224.0.0-70.239.255.255 is
# US. -KL 2012-11-27
"70.232.245.60","70.232.245.255","1189672252","1189672447","US","United States"
# GB, despite neither previous (GE) nor next (LV) MaxMind entry being GB,
# but because RIR delegation files agree with both previous and next
# MaxMind entry and say GB for 91.228.0.0-91.228.3.255. -KL 2012-11-27
"91.228.0.0","91.228.3.255","1541668864","1541669887","GB","United Kingdom"
# NL, because next MaxMind entry 176.56.173.0-176.56.173.63 is NL, and RIR
# delegation files say 176.56.160.0-176.56.191.255 is NL. -KL 2013-05-13
"176.56.172.0","176.56.172.255","2956504064","2956504319","NL","Netherlands"
# NL, despite neither previous (RU) nor next (GB) MaxMind entry being NL,
# but because RIR delegation files say entire range
# 176.56.160.0-176.56.191.255 is NL. -KL 2013-05-13
"176.56.174.0","176.56.174.255","2956504576","2956504831","NL","Netherlands"
# GB, because RIR delegation files say exactly this range
# 185.25.84.0-185.25.87.255 is GB, even though neither previous nor next
# MaxMind range is GB. Both previous and next MaxMind ranges match RIR
# delegation files, too. -KL 2013-05-13
"185.25.84.0","185.25.87.255","3105444864","3105445887","GB","United Kingdom"
# US, because next MaxMind entry 199.101.193.0-199.101.195.255 is US, and,
# together with next entries, matches RIR delegation file entry
# 199.101.192.0-199.101.199.255 which is US. -KL 2013-05-13
"199.101.192.0","199.101.192.255","3345334272","3345334527","US","United States"
# US, because ARIN says 199.255.208.0-199.255.215.255 is US.
# Changed entry start from 199.255.213.0 to 199.255.208.0 on 2013-08-12.
# Split up into 199.255.208.0-199.255.209.127 and
# 199.255.210.0-199.255.215.255 on 2013-10-11. -KL 2013-10-11
"199.255.208.0","199.255.209.127","3355430912","3355431295","US","United States"
"199.255.210.0","199.255.215.255","3355431424","3355432959","US","United States"
# EU, despite neither previous (RU) nor next (SE) MaxMind entry being EU,
# but because RIR delegation files agree with previous MaxMind entry and
# say EU for 217.15.160.0-217.15.175.255. -KL 2013-05-13
"217.15.160.0","217.15.164.255","3641679872","3641681151","EU","Europe"
# FR, because previous MaxMind entry 217.15.166.0-217.15.166.255 is FR,
# and RIR delegation files contain a block 217.15.160.0-217.15.175.255
# which, however, is EU, not FR. But merging with next MaxMind entry
# 217.15.176.0-217.15.191.255 which is KZ and which fully matches what
# the RIR delegation files say seems unlikely to be correct.
# -KL 2012-11-27
"217.15.167.0","217.15.175.255","3641681664","3641683967","FR","France"