rfcdiff inconsistently handling character escapes

[d3e3e3]

Describe the issue

Using the datatracker to compare two versions of a draft can result in bizarrely inconsistent processing of ">" escapes. In particular, if you look at any of the outputs for draft-eastlake-fnv-27 (.txt, .html, or .pdf), particularly Section 6.4 on the makefile, you will find nice ">"s and no ">"s. BUT, if you use the datatracker to diff -26 with -27, you get output for that section which seems to erroneously claim that some ">"s in Section 6.4 were not processed into ">"s!!!

Code of Conduct

• I agree to follow the IETF's Code of Conduct

[kesara]

This fallsdown to xml2rf but, xml2rfc doing the correct thing. -26 revision has <![CDATA[ ]]> around sourcecode but -27 doesn't.
So I think xml2rfc is processing the draft correctly.

[d3e3e3]

Hi, Thanks for your rapid response and sorry if my bug report was unclear. The problem, as explained in more detail below, is that IDDIFF is lying to me about version -27: Yes, -26 has a <![CDATA[ ]]>, so the escapes for < and > are not converted in -26. And in -27, the removal of the CDATA means that the escapes are converted. So yes, the output files, such as the .txt files, for -26 and -27, are correct. BUT, if you look closely, the output of IDDIFF is *false*. I have attached a screenshot. Look closely at the diffs for Section 6.4. *Very closely.* You will see that IDDIFF claims that in Section 6.4 there are some ">"s in -27 but falsely claims there are some unconverted escapes for "greater than" in -27. Yet if you look at the output (.txt, .pdf, or .html) for -27 they are all correct. Please look into this bug in IDDIFF or explain to me EXACTLY how I am wrong above. Thanks, Donald =============================== Donald E. Eastlake 3rd +1-508-333-2270 (cell) 2386 Panoramic Circle, Apopka, FL 32703 USA ***@***.***

…

On Mon, Sep 30, 2024 at 2:33 AM Kesara Rathnayake ***@***.***> wrote: This fallsdown to xml2rf but, xml2rfc doing the correct thing. -26 revision has <![CDATA[ ]]> around sourcecode but -27 doesn't. So I think xml2rfc is processing the draft correctly. — Reply to this email directly, view it on GitHub <#6>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKM6BURKDZCQOR2U2MXZE3ZZDWEXAVCNFSM6AAAAABPCQCUMGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBSGIZDMMZWGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kesara]

@d3e3e3, Your screenshot didn't come through to GitHub issues.

But I've noticed that diff output has 4 instances of "<" instead of < for draft-eastlake-fnv-26.
This is a bug in rfcdiff.
I'll move this issue to rfcdiff repository.

BTW The alternative diff tool (iddiff) seems to recognise the &lt correctly, but it is over-escaping it. ¹

Footnotes

1. https://author-tools.ietf.org/diff?doc_1=draft-eastlake-fnv-27&iddiff=1 ↩

[d3e3e3]

The bug you note in the diff output for -26 is a different bug. To see the bug I was reporting, go to the datatracker. Search for "fnv". Cick on the button to go to the "htmlized" version. On the right, select fnv-26 and fnv-27 to compare. Click on Side-by-side. The URL at the top of the resulting window including "iddiff" in the path which seems reasonable since it is comparing ids. (I do not particularly understand the difference between iddiff and rfcdiff.) In any case, examine Section 6.4. On the left, we have mostly the escapes for < and > since version -26 had a CDATA around this. On the right there should ONLY be the naked single characters "less than" and "greater than" (< and >). But, on the right, I can see at a glance there are at least three occurrences of the sequence ampersand, g, t, semicolon. (&gt;). This is WRONG. If you look at the actual .txt or .pdf or .html of version -27, they are all display correct but what looks to me like IDDIFF provides incorrect output for -27. It was only due to this incorrect output that I created and posted -28 which unnecessarily just uses ">" rather than the escape. The same bug evidences itself if you use the datatracker to compare -26 with -28.

[rjsparks]

Hi Donald - the issue here is with rfcdiff and its html generation getting html unescaping wrong - there are artifacts that are incorrect on both the left and right sides of the comparison of -26 and -27.

Specifically, these lines in -26:

   # Makefile for fnv
   # If you extract this file from RFC NNNN, the five character sequence
   #     <TAB> below must be replace with a tab (0x09) character.

render with the first < above unescaped in the left column:

and further, the corresponding lines from -27:

   # Makefile for fnv
   # If you extract this file from RFC NNNN, the five character sequence
   #     <TAB> below must be replace with a tab (0x09) character.

also render incorrectly in the diff screenshot above in the right column.

And, of course, there are other related issues elsewhere in the diff.

We've confirmed that this is actually in rfcdiff, and not in any escape handling at author-tools feeding rfcdiff. We'll work on correcting it.

Re: iddiff vs rfcdiff

rfcdiff is a shell script gluing together awk and wdiff that has been around for a long time and is difficult to make changes to (it is fragile)

iddiff is a python re-implementation of rfcdiff that is not yet bug-for-bug identical, so the older rfcdiff is used by default.

Which implementation gets used in the backend is currently controlled by the iddiff parameter - the url path always says iddiff and we understand that is confusing. Reconciling that is planned.

Again - we see the issue with rfcdiff and will work on correcting it.

[d3e3e3]

Hi Robert, Thanks. I am satisfied knowing that you agree my specific complaint is a bug and are working on it. Donald =============================== Donald E. Eastlake 3rd +1-508-333-2270 (cell) 2386 Panoramic Circle, Apopka, FL 32703 USA ***@***.***

…

On Thu, Oct 10, 2024 at 6:01 PM Robert Sparks ***@***.***> wrote: Hi Donald - the issue here is with rfcdiff and its html generation getting html unescaping wrong - there are artifacts that are incorrect on both the left and right sides of the comparison of -26 and -27. Specifically, these lines in -26: # Makefile for fnv # If you extract this file from RFC NNNN, the five character sequence # &lt;TAB&gt; below must be replace with a tab (0x09) character. render with the first &lt; above unescaped in the left column: image.png (view on web) <https://github.com/user-attachments/assets/ce9beb60-d5b8-43df-874b-af998d913037> and further, the corresponding lines from -27: # Makefile for fnv # If you extract this file from RFC NNNN, the five character sequence # <TAB> below must be replace with a tab (0x09) character. also render incorrectly in the diff screenshot above in the right column. And, of course, there are other related issues elsewhere in the diff. We've confirmed that this is actually in rfcdiff, and not in any escape handling at author-tools feeding rfcdiff. We'll work on correcting it. Re: iddiff vs rfcdiff rfcdiff is a shell script <https://github.com/ietf-tools/rfcdiff/blob/main/rfcdiff> gluing together awk and wdiff that has been around for a long time and is difficult to make changes to (it is fragile) iddiff is a python re-implementation of rfcdiff <https://github.com/ietf-tools/iddiff/blob/main/iddiff/iddiff.py> that is not yet bug-for-bug identical, so the older rfcdiff is used by default. Which implementation gets used in the backend is currently controlled by the iddiff parameter - the url path always says iddiff and we understand that is confusing. Reconciling that is planned. Again - we see the issue with rfcdiff and will work on correcting it. — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKM6BXAGVVHZB7GPGPPGRLZ232LLAVCNFSM6AAAAABPEK2X4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBWGEYTMMJVGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[chaymisanti7-jpg]

#5