Inaccurate partials proof marked as OG proof

[verifyfx]

I suspected that there are issue where partials proof were parsed

2021-07-10T02:08:40.428 harvester chia.harvester.harvester: INFO     3 plots were eligible for farming 3ff73aafb4... Found 2 proofs. Time: 2.29660 s. Total 57 plots
2021-07-10T02:08:40.447 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space

from reddit

[stolk]

Thanks for reporting. Does the dot turn cyan if you resize the terminal (w tool running) to be shorter in height?

[kastermight]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

[stolk]

Ok, I've been able to reproduce it now. And understand what is going on. This is triggered when it finds 2 proofs, it seems. And possibly only submits a partial for 1?

[stolk]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

[verifyfx]

resizjng doesn't turn it cyan

[verifyfx]

pulling latest ( 1dbc2c1 ) fixed this issue.

[kastermight]

I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D

Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log.

OK I did. My idea is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries to differentiate between OG proofs and pool proof. I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging.

[stolk]

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes?

…

On Sun, Jul 11, 2021 at 12:36 AM Kaster Might ***@***.***> wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] <https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png> Can you email your debug.log to b.stolk at gmail.com please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A> .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada ***@***.***

[verifyfx]

Found another case:
It's a case where farmer logged submitting partials before harvester logged proof found

2021-07-11T22:01:16.087 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://XXX
2021-07-11T22:01:16.406 harvester chia.harvester.harvester: INFO     2 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 1.08288 s. Total 138 plots
2021-07-11T22:01:23.747 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00621 s. Total 138 plots
2021-07-11T22:01:34.160 harvester chia.harvester.harvester: INFO     1 plots were eligible for farming 315babe1db... Found 1 proofs. Time: 2.18124 s. Total 138 plots
2021-07-11T22:01:34.203 farmer chia.farmer.farmer         : INFO     Submitting partial for XXX to https://asia1.pool.space
2021-07-11T22:01:35.091 farmer chia.farmer.farmer         : INFO     Pool response: {'new_difficulty': 4}
2021-07-11T22:01:41.452 harvester chia.harvester.harvester: INFO     0 plots were eligible for farming 315babe1db... Found 0 proofs. Time: 0.00669 s. Total 138 plots

[kastermight]

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes?

afaik they start automatically when i do chia start farmer.

[stolk]

Those are threads of a single process. A thread per peer connection, probably. On Sun, Jul 11, 2021 at 8:56 AM Kaster Might ***@***.***> wrote:

Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes? … <#m_-1223658502728578559_> On Sun, Jul 11, 2021 at 12:36 AM Kaster Might *@*. *> wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png <https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png> Can you email your debug.log to b.stolk at gmail.com <http://gmail.com> please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment) <#42 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A <https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A> . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.* afaik they start automatically when i do chia start farmer. [image: Screenshot from 2021-07-11 08-56-13] <https://user-images.githubusercontent.com/2565943/125201878-e2b34980-e225-11eb-8862-e5bb5eb8b01d.png> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADPE4AFQQMYTCBUAEXLTIDTXG5MHANCNFSM5AEPJL7A> .

-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada ***@***.***

[kastermight]

Those are threads of a single process. A thread per peer connection, probably. On Sun, Jul 11, 2021 at 8:56 AM Kaster Might @.> wrote:
Huh? that makes absolutely no sense, to run 48 full nodes on one machine. Are you sure you actually have that many running? How would you even start that many nodes? … <#m_-1223658502728578559_> On Sun, Jul 11, 2021 at 12:36 AM Kaster Might @. > wrote: I have those as well. Too many in fact. Not paying too much attention though, because I know those are not OG proofs by looking at my shiny empty wallet :D [image: Screenshot from 2021-07-10 12-16-00] https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png https://user-images.githubusercontent.com/2565943/125174231-9eb63b00-e178-11eb-97ad-b0416e935f93.png Can you email your debug.log to b.stolk at gmail.com http://gmail.com please? I would like to test a fix on your log. OK I did. My ideas is there's some race condition going on since I have 48 full_nodes running (number of my cores) and sometimes they write in off order. I can see that you check for "Submitting partials" entries after "Found n proofs" entries, and I noticed that sometimes "submitting" entries come earlier than the "found" entries. But I might be wrong, so good luck with debugging. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment) <#42 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A https://github.com/notifications/unsubscribe-auth/AADPE4E5VMX75S2MAJXBDRLTXFCZXANCNFSM5AEPJL7A . -- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @. afaik they start automatically when i do chia start farmer. [image: Screenshot from 2021-07-11 08-56-13] https://user-images.githubusercontent.com/2565943/125201878-e2b34980-e225-11eb-8862-e5bb5eb8b01d.png — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPE4AFQQMYTCBUAEXLTIDTXG5MHANCNFSM5AEPJL7A .
-- Owner/Director of Game Studio Abraham Stolk Inc. Vancouver BC, Canada @.

Hi. Those are not threads as they have separate PIDs and TGIDs. I dont' think it matters though.

I saw your email, but I still can see blue squares. Maybe you had different branch or something. I'm on latest master. Recompiled and everything. Attaching both screenshots and the log file here just in case.

PS: I noticed that if the terminal window is short enough, blue squares disappear. Maybe that's what you're seeing.

debug.log

[xordos]

I want share my log as I have simplest (single machine run full farmer) case and I do see the problem too.

Basically, in the log, sometime, the "Submitting partial" log line doesn't immediate after the eligible line. I have 2 such cases that show blue dot but are actually pool proof. I think this is a race condition that these 2 lines are from different process. So the order is not guaranteed, see the logging process name (farmer vs harvester) below:

2021-07-12T16:42:04.397 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.35231 s. Total 798 plots
2021-07-12T16:42:14.216 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX
2021-07-12T16:42:14.299 harvester chia.harvester.harvester: INFO 4 plots were eligible for farming 686599092a... Found 1 proofs. Time: 1.75216 s. Total 798 plots
2021-07-12T16:42:14.398 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1}
2021-07-12T16:42:22.282 harvester chia.harvester.harvester: INFO 2 plots were eligible for farming 686599092a... Found 0 proofs. Time: 0.06132 s. Total 798 plots

2021-07-13T06:56:25.522 harvester chia.harvester.harvester: INFO 1 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.20828 s. Total 799 plots
2021-07-13T06:56:34.317 farmer chia.farmer.farmer : INFO Submitting partial for 10768b8bd5ab299b85a2c3d8fc80f1a70e03b898acd8292aeb7eb959a2765c04 to https://XXX
2021-07-13T06:56:34.357 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 1 proofs. Time: 1.09751 s. Total 799 plots
2021-07-13T06:56:34.523 farmer chia.farmer.farmer : INFO Pool response: {'new_difficulty': 1}
2021-07-13T06:56:43.013 harvester chia.harvester.harvester: INFO 3 plots were eligible for farming 4c1214cc87... Found 0 proofs. Time: 0.12265 s. Total 799 plots

BTW, your tool is really useful and thanks for your hard work!!