PixivFE/doc/dev/features/i18n.md

Internationalization

!!! warning Important notice: please don't translate server logs with i18n.*(). It uses the request locale, not the server one.

Quick links to Crowdin

• Project Dashboard
• Project Settings
• Translation Editor
• API Key

Todo

• Add cookie: Locale
• Rewrite jet template on load to use translated strings
• Add dedicated option to set locale in settings page
• Check if any jet template strings are ignored (false negative)
• Setup Crowdin
  • manual upload and download
  • automatic upload and download
  • automated upload and download via Woodpecker CI

Crowdin CLI usage

Install the CLI: See official instructions

Remember to check crowdin -V. Should be 4.2.0 or later.

First, set up API token:

1. Go to https://crowdin.com/settings#api-key
2. Click the "New Token" button
3. Set permission: Projects >
   • Projects (List, Get, Create, Edit) -- Read only
   • Source files & strings (List, Get, Create, Edit) -- Read and write
   • Translations (List, Get, Create, Edit) -- Read and write
4. Copy the new token and save it somewhere

Then, try it out.

export CROWDIN_PERSONAL_TOKEN=token_here # put this somewhere in your shell config, or the `.env` file inside this repo, which will be used by ./build.sh
crowdin upload
crowdin download

Crowdin Docker usage

These instructions assume CROWDIN_PERSONAL_TOKEN is available as an environment variable.

Step 0: Generate i18n source files

Create locale/en source files (code.json and template.json):

./build.sh i18n

Step 1: Set up Docker container

Create an interactive container using crowdin/cli:latest, mounting local files:

docker run -it --rm --name crowdin-cli \
  -e CROWDIN_PERSONAL_TOKEN=${CROWDIN_PERSONAL_TOKEN} \
  -v "./i18n/locale:/usr/crowdin-project/i18n/locale" \
  -v "./crowdin.yml:/usr/crowdin-project/crowdin.yml" \
  crowdin/cli:latest

Step 2: Upload local sources to Crowdin

Update locale/en sources on the Crowdin project:

crowdin upload --verbose

Step 3: Download translations to local repository

Fetch new translations from Crowdin and update non-English locales:

crowdin download --verbose

Crowdin Web UI usage

Add new languages to translate in Settings > Languages.

./i18n/crawler -- Primer on html and jet's treatment of templates

<a>abc {{.Hi}}</a>

html: abc {{.Hi}} is text

jet: <a>abc is text jet: </a> is text

So, jet doesn't care about HTML.

And we have to care about {* *} comments.

./i18n/crawler -- Coalesce inline tags

Example: in below, the <a> tag should be part of the string.

Log in with your Pixiv account's cookie to access features above. To learn how to obtain your cookie, please see <a href="https://pixivfe-docs.pages.dev/obtaining-pixivfe-token/">the guide on obtaining your PixivFE token</a>.

Tags to consider as inline: a

Implementation details

!!! warning This section was written using an LLM with codebase context, and may contain inaccuracies

Should be used as a rough primer on the i18n implementation only

This section covers implementation details for the i18n system.

The i18n system consists of the following components:

Component	Description	Key files/functions
Crawler	Extracts translatable strings from HTML template files	crawler/main.go
Converter	Processes crawler output to generate translation map	converter/main.go, i18n.SuccintId()
Locale files	Store en source and translations in JSON format	i18n/locale/<lang_code>/code.json, i18n/locale/<lang_code>/template.json
Lookup and rewrite functions	Core i18n functionality for loading translations and looking up strings	lookup.go, rewrite.go
Integration	Wrapper functions for automatic translation lookup	Tr(), Sprintf()

Additional notes:

• Uses xxHash for string hashing when generating IDs
• Caches strings.Replacer objects for performance
• Supports different locales per goroutine using routine.InheritableThreadLocal
• Includes a Semgrep rule (semgrep-i18n.yml) for detecting untranslated strings

1. Crawler

The crawler (crawler/main.go) scans HTML template files to extract translatable strings.

• Uses the html package to parse HTML
• Traverses the DOM tree to find text nodes
• Ignores certain patterns (e.g., Jet template commands and strings included in the IgnoreTheseStrings variable)
• Outputs a JSON array of objects containing the message and file path

Example output:

[
  {
    "msg": "Translatable string",
    "file": "path/to/file.html"
  }
]

2. Converter

The converter (converter/main.go) processes the crawler output to generate a translation map.

• Reads the crawler JSON from stdin
• Generates a unique ID for each string using i18n.SuccintId()
• Outputs a JSON object mapping IDs to original strings

Example output:

{
  "path/to/file.html:uniqueId": "Translatable string"
}

3. Locale files

Translations are stored in JSON files under i18n/locale/<lang_code>/:

• code.json: Translations for strings in Go code
• template.json: Translations for strings in HTML templates

The base locale (English) contains the original strings, while other locales contain translated strings.

4. Lookup and rewrite functions

The core i18n functionality is implemented in lookup.go and rewrite.go.

4.1. Lookup

lookup.go provides functions to load translations and look up strings:

• Init(): Loads all locale files into memory
• __lookup_skip_stack_2(): Performs the actual string lookup
• SuccintId(): Generates a unique ID for a string based on file path and content

4.2. Rewrite

rewrite.go handles template rewriting:

• Replacer(): Returns a strings.Replacer for a given locale and file
• translationPairs_inner(): Generates replacement pairs for a locale and file

5. Integration

The i18n system is integrated into the application code using wrapper functions that automatically look up translations based on the current locale:

func Tr(text string) string {
    return __lookup_skip_stack_2(GetLocale(), text)
}

func Sprintf(format string, a ...any) string {
    format = __lookup_skip_stack_2(GetLocale(), format)
    return fmt.Sprintf(format, a...)
}