docs: i18n implementation details

2024-12-06 19:16:23 +01:00 · 2024-10-11 16:03:41 +11:00
parent 9f5faa7cf8
commit bedb5eba6b
1 changed files with 118 additions and 11 deletions
@@ -1,6 +1,7 @@
-# Localization
+# Internationalization

-Important notice: please don't translate server logs with `i18n.*()`. It uses the request locale, not the server one.
+!!! warning
+    **Important notice:** please don't translate server logs with `i18n.*()`. It uses the request locale, not the server one.

 ## Quick links to Crowdin

@@ -15,9 +16,10 @@ Important notice: please don't translate server logs with `i18n.*()`. It uses th
 - [x] Rewrite jet template on load to use translated strings
 - [ ] Add dedicated option to set locale in settings page
 - [ ] Check if any jet template strings are ignored (false negative)
- Setup Crowdin
-  - [x] manual upload and download
-  - [x] automatic upload and download
+- [x] Setup Crowdin
+    - [x] manual upload and download
+    - [x] automatic upload and download
+    - [ ] automated upload and download via Woodpecker CI

 ## Crowdin CLI usage

@@ -25,19 +27,19 @@ Install the CLI: [See official instructions](https://crowdin.github.io/crowdin-c

 Remember to check `crowdin -V`. Should be `4.2.0` or later.

-First, setup API token:
+First, set up API token:

-1. Go to https://crowdin.com/settings#api-key
+1. Go to [https://crowdin.com/settings#api-key](https://crowdin.com/settings#api-key)
 2. Click the "New Token" button
 3. Set permission: Projects >
-  - Projects (List, Get, Create, Edit) -- Read only
-  - Source files & strings (List, Get, Create, Edit) -- Read and write
-  - Translations (List, Get, Create, Edit) -- Read and write
+    - Projects (List, Get, Create, Edit) -- Read only
+    - Source files & strings (List, Get, Create, Edit) -- Read and write
+    - Translations (List, Get, Create, Edit) -- Read and write
 4. Copy the new token and save it somewhere

 Then, try it out.

-```shell
+```sh
 export CROWDIN_PERSONAL_TOKEN=token_here # put this somewhere in your shell config, or the `.env` file inside this repo, which will be used by ./build.sh
 crowdin upload
 crowdin download
@@ -111,3 +113,108 @@ Log in with your Pixiv account's cookie to access features above. To learn how t
 ```

 Tags to consider as inline: `a`
+
+## Implementation details
+
+!!! warning
+    This section was written using an LLM with codebase context, and may contain inaccuracies
+
+    Should be used as a rough primer on the i18n implementation only
+
+This section covers implementation details for the i18n system.
+
+The i18n system consists of the following components:
+
+| Component | Description | Key files/functions |
+|-----------|-------------|---------------------|
+| Crawler | Extracts translatable strings from HTML template files | `crawler/main.go` |
+| Converter | Processes crawler output to generate translation map | `converter/main.go`, `i18n.SuccintId()` |
+| Locale files | Store `en` source and translations in JSON format | `i18n/locale/<lang_code>/code.json`, `i18n/locale/<lang_code>/template.json` |
+| Lookup and rewrite functions | Core i18n functionality for loading translations and looking up strings | `lookup.go`, `rewrite.go` |
+| Integration | Wrapper functions for automatic translation lookup | `Tr()`, `Sprintf()` |
+
+Additional notes:
+
+- Uses `xxHash` for string hashing when generating IDs
+- Caches `strings.Replacer` objects for performance
+- Supports different locales per goroutine using `routine.InheritableThreadLocal`
+- Includes a Semgrep rule (`semgrep-i18n.yml`) for detecting untranslated strings
+
+### 1. Crawler
+
+The crawler (`crawler/main.go`) scans HTML template files to extract translatable strings.
+
+- Uses the `html` package to parse HTML
+- Traverses the DOM tree to find text nodes
+- Ignores certain patterns (e.g., Jet template commands and strings included in the `IgnoreTheseStrings` variable)
+- Outputs a JSON array of objects containing the message and file path
+
+**Example output:**
+
+```json
+[
+  {
+    "msg": "Translatable string",
+    "file": "path/to/file.html"
+  }
+]
+```
+
+### 2. Converter
+
+The converter (`converter/main.go`) processes the crawler output to generate a translation map.
+
+- Reads the crawler JSON from stdin
+- Generates a unique ID for each string using `i18n.SuccintId()`
+- Outputs a JSON object mapping IDs to original strings
+
+**Example output:**
+
+```json
+{
+  "path/to/file.html:uniqueId": "Translatable string"
+}
+```
+
+### 3. Locale files
+
+Translations are stored in JSON files under `i18n/locale/<lang_code>/`:
+
+- `code.json`: Translations for strings in Go code
+- `template.json`: Translations for strings in HTML templates
+
+The base locale (English) contains the original strings, while other locales contain translated strings.
+
+### 4. Lookup and rewrite functions
+
+The core i18n functionality is implemented in `lookup.go` and `rewrite.go`.
+
+#### 4.1. Lookup
+
+`lookup.go` provides functions to load translations and look up strings:
+
+- `Init()`: Loads all locale files into memory
+- `__lookup_skip_stack_2()`: Performs the actual string lookup
+- `SuccintId()`: Generates a unique ID for a string based on file path and content
+
+#### 4.2. Rewrite
+
+`rewrite.go` handles template rewriting:
+
+- `Replacer()`: Returns a `strings.Replacer` for a given locale and file
+- `translationPairs_inner()`: Generates replacement pairs for a locale and file
+
+### 5. Integration
+
+The i18n system is integrated into the application code using wrapper functions that automatically look up translations based on the current locale:
+
+```go
+func Tr(text string) string {
+    return __lookup_skip_stack_2(GetLocale(), text)
+}
+
+func Sprintf(format string, a ...any) string {
+    format = __lookup_skip_stack_2(GetLocale(), format)
+    return fmt.Sprintf(format, a...)
+}
+```