- Published on
How to Detect File Encoding Before Converting TXT to UTF-8
When a TXT file looks broken, the first question is not always "How do I convert it?" Sometimes the better first question is: What encoding is this file using?
A file encoding detector tries to answer that question by looking at the raw bytes in the file. The result is useful, but it is not magic. Short files, mixed-language files, and files that have already been damaged can be hard to classify. The safest workflow is to detect, preview, manually test if needed, and only then download a UTF-8 copy.
You can start with the WristTale TXT encoding converter. It detects a likely source encoding, lets you switch manually, previews the converted text, and downloads a UTF-8 file locally in your browser.
What a file encoding detector can and cannot know
Plain .txt files usually do not include reliable metadata that says "this is GBK" or "this is Big5." A detector has to infer the answer from byte patterns.
It can often identify:
- UTF-8 with or without a byte order mark.
- UTF-16 LE or UTF-16 BE when the file has a BOM or strong alternating null-byte pattern.
- GB-family Chinese encodings when the byte distribution looks like simplified Chinese text.
- Big5 when the byte distribution looks like traditional Chinese text.
- Shift_JIS or EUC-JP when Japanese kana and CJK byte patterns are likely.
- EUC-KR when Korean byte patterns are likely.
It cannot guarantee the correct answer for every file. Encoding detection is especially uncertain when a file is very short, mostly ASCII, mixed from multiple sources, or already saved after mojibake appeared.
Start with BOM and UTF-16 clues
A byte order mark, or BOM, is the strongest early clue. Unicode describes BOM as a signature at the beginning of a stream that can identify a Unicode encoding form. For UTF-16, it also tells the byte order.
Common BOM clues:
| Bytes at file start | Likely encoding |
|---|---|
EF BB BF | UTF-8 with BOM |
FF FE | UTF-16 LE |
FE FF | UTF-16 BE |
UTF-8 does not have byte-order ambiguity, so a UTF-8 BOM is only a signature. It can still help some Windows tools identify plain text as UTF-8.
If there is no BOM, UTF-16 can still be detectable. English-heavy UTF-16 often has many zero bytes in every other byte position. That pattern is a strong hint that the file is UTF-16 LE or UTF-16 BE.
Then test whether the file is valid UTF-8
UTF-8 has strict byte rules. If a byte sequence is invalid UTF-8, the file is probably a legacy encoding such as GBK, Big5, Shift_JIS, EUC-JP, EUC-KR, or Windows-1252.
If the file is valid UTF-8, the detector should normally keep it as UTF-8. A file containing only basic English characters may also be valid ASCII, but UTF-8 is still the right modern target format because ASCII text is compatible with UTF-8.
Use language and source as practical clues
If automatic detection is uncertain, use the file's likely origin.
| Clue | Encoding to try |
|---|---|
| Simplified Chinese TXT novel from an older Windows source | GB18030, then GBK |
| Traditional Chinese text from Taiwan or Hong Kong | Big5 |
| Japanese TXT archive or old game script | Shift_JIS, then EUC-JP |
| Korean plain text | EUC-KR |
| Western Windows notes with curly quotes | Windows-1252 |
| Text has many empty-looking gaps or odd spacing | UTF-16 LE or UTF-16 BE |
For Chinese TXT novels, GB18030 is a strong manual fallback because it covers a broad GB-family range. If GB18030 gives readable text, you usually do not need to test GBK or GB2312 separately.
Preview before you download
The preview is the real quality check. A detector can choose an encoding, but you decide whether the output is readable.
Before downloading the UTF-8 copy, scan:
- The first chapter title.
- A paragraph with punctuation.
- Any names or rare characters.
- The middle of the file, not only the beginning.
- The end of the file, especially if the TXT was stitched from multiple sources.
If the beginning is readable but the middle breaks, the file may contain mixed encodings or inserted boilerplate. Try converting the original file again with a different source encoding, or split the file around the broken section.
How this helps WristTale imports
WristTale works best with UTF-8 text. When the text is decoded correctly before import, the app has a better chance of recognizing chapter headings, showing the same text on phone and watch, and syncing clean chunks to a Garmin watch.
If you import a file while it is still garbled, chapter detection may fail even when the original chapter titles were normal. Fix the encoding first, then review chapters.
Recommended workflow:
- Detect the source encoding in the TXT encoding converter.
- Preview the converted text.
- Download the UTF-8 copy only when the preview is readable.
- Import the UTF-8 file into WristTale.
- Check chapter detection before syncing to the watch.
If the text is already visibly garbled, read the garbled Chinese text guide for a more problem-focused recovery workflow.
Privacy note
Encoding detection does not require uploading the file. The WristTale converter reads the selected TXT file with browser APIs and performs detection and conversion on the current device. This is the right default for personal notes, legally owned ebooks, study material, and files that should not leave your computer.
References
- WHATWG Encoding Standard defines the web platform encoding names and JavaScript decoding API used by browsers.
- MDN TextDecoder documents the browser API for decoding bytes into strings with a selected encoding.
- Unicode UTF-8, UTF-16, UTF-32 & BOM FAQ explains how BOM signatures work for Unicode text streams.