diff --git a/doc/dev/features/i18n.md b/doc/dev/features/i18n.md index adc5600..c6ed0d6 100644 --- a/doc/dev/features/i18n.md +++ b/doc/dev/features/i18n.md @@ -1,6 +1,7 @@ -# Localization +# Internationalization -Important notice: please don't translate server logs with `i18n.*()`. It uses the request locale, not the server one. +!!! warning + **Important notice:** please don't translate server logs with `i18n.*()`. It uses the request locale, not the server one. ## Quick links to Crowdin @@ -15,9 +16,10 @@ Important notice: please don't translate server logs with `i18n.*()`. It uses th - [x] Rewrite jet template on load to use translated strings - [ ] Add dedicated option to set locale in settings page - [ ] Check if any jet template strings are ignored (false negative) -- Setup Crowdin - - [x] manual upload and download - - [x] automatic upload and download +- [x] Setup Crowdin + - [x] manual upload and download + - [x] automatic upload and download + - [ ] automated upload and download via Woodpecker CI ## Crowdin CLI usage @@ -25,19 +27,19 @@ Install the CLI: [See official instructions](https://crowdin.github.io/crowdin-c Remember to check `crowdin -V`. Should be `4.2.0` or later. -First, setup API token: +First, set up API token: -1. Go to https://crowdin.com/settings#api-key +1. Go to [https://crowdin.com/settings#api-key](https://crowdin.com/settings#api-key) 2. Click the "New Token" button 3. Set permission: Projects > - - Projects (List, Get, Create, Edit) -- Read only - - Source files & strings (List, Get, Create, Edit) -- Read and write - - Translations (List, Get, Create, Edit) -- Read and write + - Projects (List, Get, Create, Edit) -- Read only + - Source files & strings (List, Get, Create, Edit) -- Read and write + - Translations (List, Get, Create, Edit) -- Read and write 4. Copy the new token and save it somewhere Then, try it out. -```shell +```sh export CROWDIN_PERSONAL_TOKEN=token_here # put this somewhere in your shell config, or the `.env` file inside this repo, which will be used by ./build.sh crowdin upload crowdin download @@ -111,3 +113,108 @@ Log in with your Pixiv account's cookie to access features above. To learn how t ``` Tags to consider as inline: `a` + +## Implementation details + +!!! warning + This section was written using an LLM with codebase context, and may contain inaccuracies + + Should be used as a rough primer on the i18n implementation only + +This section covers implementation details for the i18n system. + +The i18n system consists of the following components: + +| Component | Description | Key files/functions | +|-----------|-------------|---------------------| +| Crawler | Extracts translatable strings from HTML template files | `crawler/main.go` | +| Converter | Processes crawler output to generate translation map | `converter/main.go`, `i18n.SuccintId()` | +| Locale files | Store `en` source and translations in JSON format | `i18n/locale//code.json`, `i18n/locale//template.json` | +| Lookup and rewrite functions | Core i18n functionality for loading translations and looking up strings | `lookup.go`, `rewrite.go` | +| Integration | Wrapper functions for automatic translation lookup | `Tr()`, `Sprintf()` | + +Additional notes: + +- Uses `xxHash` for string hashing when generating IDs +- Caches `strings.Replacer` objects for performance +- Supports different locales per goroutine using `routine.InheritableThreadLocal` +- Includes a Semgrep rule (`semgrep-i18n.yml`) for detecting untranslated strings + +### 1. Crawler + +The crawler (`crawler/main.go`) scans HTML template files to extract translatable strings. + +- Uses the `html` package to parse HTML +- Traverses the DOM tree to find text nodes +- Ignores certain patterns (e.g., Jet template commands and strings included in the `IgnoreTheseStrings` variable) +- Outputs a JSON array of objects containing the message and file path + +**Example output:** + +```json +[ + { + "msg": "Translatable string", + "file": "path/to/file.html" + } +] +``` + +### 2. Converter + +The converter (`converter/main.go`) processes the crawler output to generate a translation map. + +- Reads the crawler JSON from stdin +- Generates a unique ID for each string using `i18n.SuccintId()` +- Outputs a JSON object mapping IDs to original strings + +**Example output:** + +```json +{ + "path/to/file.html:uniqueId": "Translatable string" +} +``` + +### 3. Locale files + +Translations are stored in JSON files under `i18n/locale//`: + +- `code.json`: Translations for strings in Go code +- `template.json`: Translations for strings in HTML templates + +The base locale (English) contains the original strings, while other locales contain translated strings. + +### 4. Lookup and rewrite functions + +The core i18n functionality is implemented in `lookup.go` and `rewrite.go`. + +#### 4.1. Lookup + +`lookup.go` provides functions to load translations and look up strings: + +- `Init()`: Loads all locale files into memory +- `__lookup_skip_stack_2()`: Performs the actual string lookup +- `SuccintId()`: Generates a unique ID for a string based on file path and content + +#### 4.2. Rewrite + +`rewrite.go` handles template rewriting: + +- `Replacer()`: Returns a `strings.Replacer` for a given locale and file +- `translationPairs_inner()`: Generates replacement pairs for a locale and file + +### 5. Integration + +The i18n system is integrated into the application code using wrapper functions that automatically look up translations based on the current locale: + +```go +func Tr(text string) string { + return __lookup_skip_stack_2(GetLocale(), text) +} + +func Sprintf(format string, a ...any) string { + format = __lookup_skip_stack_2(GetLocale(), format) + return fmt.Sprintf(format, a...) +} +```