some text files have BOMs other don't ?

[eshellman]

78351 for example

[eshellman]

So, this brings up the question of Unitame, which is some sort of old, old ppgen encoding stuff https://www.pgdp.net/wiki/PPTools/Ppgen/Manual#:~:text=It%20basically%20implements%20the%20standard,Sometimes%2C%20however%2C%20you%20will

in Ebookmaker/writers/TxtWriter.py, there is a line:
data = data.encode('utf_8_sig' if encoding == 'utf-8' else encoding, 'unitame')
The utf_8_sig generates UTF-8 with BOM. for plain UTF-8 we would just remove the line
It seems PG is some sort of custom PG encoding designed for a world without Unicode.

DO WE NEED TO KEEP Ebookmaker/Unitame.py ???

Do we need the BOM? If our server is always setting the utf-8 mime type, we don't need it. Probably we didn't use to set that MIME type because the text files had a mixture of encodings. Now, we should work to make sure everything is unicode.

[eshellman]

maybe @srjfoo or @70ray may know something?

[rtonsing]

I think the guys above are authoritative. The main tool now for DP is Guiguts 2, which will not save as anything but plain UTF-8 w/o BOM.
PG policy for years now has been UTF-8 w/o BOM. I remove BOM from any old book that I edit, and convert to UTF-8 if there is none. IMHO, if anything is calling this to generate old formats, it should be stopped. Replace with a stub to sound a siren.

[srjfoo]

My understanding is that, while a BOM is okay in UTF-8, it's not really necessary (or perhaps even desirable, though I don't know this for sure). Yes, it's absolutely necessary for UTF-16 and UTF-32.

I'd verify with @cpeel, @windymilla and @tangledhelix

Edit: gg2 saves as UTF-8 without BOM.

[cpeel]

Looping @wf49670 into this convo as the ppgen maintainer.

The DP wiki just says that ppgen uses the same unitame encoding as PG for Latin-1 texts and the ppgen manual says:

The generation of a Latin-1 file is provided for legacy requirements and software, but only when the PPer explicitly requests it using the "-ol" command line parameter.

I don't think PG needs to retain unitame support.

[tangledhelix]

BOM is something I don't know much about. Reading what others have said, it certainly seems to me as if we don't need or want BOM.

FWIW, the DP wiki has a page on BOM (not part of official documentation) that says BOM needs to be removed before upload to SR, PPV, or PG.

[windymilla]

Broadly agreeing with everyone else. To the best of my knowledge:

1. Guiguts 2 (and any recent Guiguts 1) does not include a BOM when saving a UTF-8 file.
2. When loading a file (for Content Providing or PPing purposes) any BOM detected on the first line is removed. When PPing, a warning is issued if this situation occurs.
3. DP expectation is that BOM is not included in text or HTML files when uploaded to DP or to PG.
4. No-one should be using the "-ol" parameter to ppgen.
5. Ebookmaker should not need Ebookmaker/Unitame.py.

[eshellman]

Thank you everyone! I will proceed with removing Unitame and leaving the txt files BOM-less.