Three user stories: How does wallet3 address those?

[spirobel]

• An exchange backend / payment processor written in golang

The exchange backend is written in golang and needs to sync many wallets at once. The wallets are view only. The goal is to fetch the blocks only once from the node (that is hosted on a different server) and use the data to sync many different wallets in parallel. The Monero related code should be directly integrated into the go binaries for ease of deployment and maintenance.

• A decentralized exchange developed by a community funded team:

The team decided to target wasm because it means the code will be platform independent. In this way they can rest assured their code will run on many different operating systems like Mac, Windows and Linux. So even with a small team they are able to service a wide range of users on many different systems. They can focus most of their development efforts on crafting a great UX, because they dont run into many build issues and platform specific bugs. Their protocol also requires the simultaneous syncing of many public view only wallets.
While wasm is a single threaded target without garbage collection (the memory can only grow). They never run into memory overflow issues because they have very small pieces of wasm that get orchestrated by a javascript codebase (that uses webworkers for threading and fetch to retrieve the blocks from the daemon)

• A mobile wallet written in flutter

A widely used mobile wallet that focuses on reliability and an easy to use user interface. Many users depend on their software so the most important goal is having a robust architecture. They take extra care to make sure there are no concurrency bugs in the wallet. They don't want the user interface to ever become unresponsive under any circumstances. Even if there is a connection loss or the power saving features of the mobile operating system kick in: the user experience should not be degraded.

How does wallet3 address these user stories?
Which API calls would the respective developer use?
How can all of these different use cases with different targets and languages be addressed without a C ABI?

[sausagenoods]

I maintain a payment processor in Golang. It's not possible to link with C++. Currently we have to deploy monero-wallet-rpc server to use wallet functions but it sometimes let's us down when it comes to availability and threading. It is also harder for the user to get monero-wallet-rpc running. What would be ideal for our project is a library that safely implements wallet functions in C.

This linking problem isn't unique to Golang. Rust, C and other languages also do not support calling C++ but they do support C.

[rbrunner7]

I go into speculation mode:

I speculate you are convinced that it's possible to build what you call here wallet3 in a way that turns all your 3 user stories and possibly many more into a pleasure, because more or less whatever people want to implement "on top" of Monero, those people find that this wallet3 thing just works for them, without much friction, and without the need for much contortions or even hacks.

I say that now as only one of multiple members of that "Seraphis wallet workgroup", my personal opinion as a Monero dev: I am pretty sure it's not possible to build something like that. You can't get such wildly differing requirements like you lay them out here with your 3 user stories all under a single hat, if only you are clever and build everything in a certain way.

Maybe, just maybe, it would be possible if we could rewrite the whole Monero codebase again from scratch, and if we had the time and the resources to do it, and some world-class devs to pull it through. All things we can't do and don't have.

What you wrote so far did not convince me to propose to stop what we are doing in that workgroup, go back to the drawing board, re-group and re-orient. But of course I am only one of the members of that workgroup, not even one with an official title, as you correctly noticed. My personal opinion.

Nothing is lost then, this second issue and the first one are open and public, and if you manage to sway the "loose consensus" of the workgroup about what we should build, and how we should build it, no problem whatsoever. I would be ready to also manage a workgroup with a quite different approach, if it still serves the goal of "No wallet left behind", the goal that the hardfork to Seraphis won't decimate the array of software that people can use to manage their XMR.

[sneurlax]

I am a mobile wallet written in flutter user/developer and will be involved in implementing Seraphis testnet in Dart for Flutter and will contribute input to the project with regards to how Serpahis/wallet3 enables or hinders that usecase

[j-berman]

On 1:

The exchange backend is written in golang and needs to sync many wallets at once. The wallets are view only.

Today this particular use case is satisfied by a light wallet server. Light wallet servers sync many view only wallets at once, then return data necessary to calculate balance to respective clients.

Seraphis/Jamtis enable a light wallet server with stronger privacy protections that similarly would be able to return data necessary to calculate balance to respective clients.

On the server: it remains an open question if a light wallet server implementation will be included in the reference monero binaries or shipped/maintained separately as they are today (see monero-lws, openmonero).

On the client: @UkoeHB has written wrapper logic to sync and calculate balance using sync'd data returned by a Seraphis compatible light wallet server in his seraphis lib. A wallet3 should be able to use that.

[spirobel]

On the server: it remains an open question if a light wallet server implementation will be included in the reference monero binaries or shipped/maintained separately as they are today (see monero-lws, openmonero).

Implementing a light wallet server is a good example. It is similar to story 1 and 2. The syncing function will be different from what we would typically expect from a mobile wallet for example. It should be possible to sync many wallets at once with the data supplied by one daemon connection.

So the API would need to provide a function for processing one block at a time. And there needs to be a clear description of what the syncing algorithm needs to do in the case of reorgs. We need a detailed description of what to look out for there. Or do you think it would be possible to provide a helper function that takes care of the pitfalls and can encapsulate this piece of complexity?

Because there are so many different circumstances under which wallet software needs to run, it seems like the syncing functions should not be hard coded in the library. It should be modular enough so that the different wallets can adapt to their use cases.

[j-berman]

Or do you think it would be possible to provide a helper function that takes care of the pitfalls and can encapsulate this piece of complexity?

@UkoeHB has written helper functions that handle reorgs here: https://github.com/UkoeHB/monero/blob/99b15c2a2c29c7a4163bbbe4512aa451a5edb776/src/seraphis/tx_enote_scanning.cpp

It should be modular enough so that the different wallets can adapt to their use cases.

These functions seem to be written with the modularity you want (and I agree that they would ideally have) in mind. Example: notice how process_ledger_for_full_refresh_onchain_pass returns ScanStatus, which indicates to the caller how they should proceed if the scanner detects a reorg.

[spirobel]

These functions seem to be written with the modularity you want (and I agree that they would ideally have) in mind.

I am not so sure about this. Here is the part in the presentation where this topic is covered: https://www.youtube.com/watch?v=aAvSpfll9z4&t=6203s It seems like everything is tightly coupled and even networking and concurrency is interwoven into the code.

I also sincerely don't understand the aversion to providing a C ABI to this. As mentioned earlier all the other languages can not interface with CPP. So they need to implement custom wrappers and make their own build processes for the Monero codebase.
All of this increases software supply chain security risk unnecessarily. And makes life for wallet developers harder.

[UkoeHB]

I also sincerely don't understand the aversion to providing a C ABI to this.

All of this is work - work for C++ devs who don't actually like C (at least I don't). You are welcome to propose a rewrite that is pure C. I'd actually be interested to see an alternative to the balance recovery framework currently used, because I don't know a cleaner way to do it that is equally generic - would be great to learn a new way.

[spirobel]

All of this is work - work for C++ devs who don't actually like C (at least I don't).

This is not about what you like or not. This is about what is needed: a C ABI.

Not providing one means that every single project needs to have a custom build of Monero and create this extern "C" wrapper that you don't want to provide.

Here is cake wallet for example: https://github.com/cake-tech/cake_wallet/blob/50fd6accef4cdc63e7f8a172129bfad734110f18/cw_monero/ios/Classes/monero_api.h#L7

[UkoeHB]

This is not about what you like or not. This is about what is needed: a C ABI.

Sure it is, I have been doing what I like for years and don't plan to stop any time soon :). In any case, if you want a C ABI then by all means go for it. Or perhaps @j-berman in the course of his work will see the bigger picture of how everything fits together into the ecosystem, and will come up with a nice C interface.

[spirobel]

Sure it is, I have been doing what I like for years and don't plan to stop any time soon :).

You get paid by this project to solve a problem. This is not your personal pet project "to have fun". I am astonished that you want to move a project with a 2 billion dollar marketcap to a new unproven protocol with this attitude.

[UkoeHB]

Is having fun mutually exclusive with improving a major cryptocurrency? Please, instead of this drama and trying to boss me around (??), go review all the seraphis material and find some specific low-quality thing I produced, some example of me disrespecting Monero core design principles, whatever you like.

[sneurlax]

@spirobel there are teams of researchers outside of Koe that set the direction of development. This isn't code done on a lark

[j-berman]

Because there are so many different circumstances under which wallet software needs to run, it seems like the syncing functions should not be hard coded in the library. It should be modular enough so that the different wallets can adapt to their use cases.

Now that I've had some time to work with the lib, I can say with certainty it is written modularly and can be adapted to various use cases.

Check out this code I wrote: https://github.com/j-berman/monero-cpp/blob/9f788b40c60d469381d8b9790a0d50aee917f59e/test/scratchpad.cpp#L27-L73

It's using the Seraphis lib to identify a user's spends and receives, reading a mock response to /getblocks.bin. Because of how cleanly the Seraphis lib's code is written, it's relatively simple to plug and play its various functions to accomplish this. The lib does not handle network requests, nor does it handle threading. I can see pretty clearly how to optimally structure my calling code to achieve performance on par with (or perhaps even better than) wallet2, and how to easily implement an efficient multi-wallet view-only scanner using the lib.

Notice how with the Seraphis lib, there is a scanning util sp::try_find_legacy_enotes_in_tx that identifies received enotes per tx, and then expects the caller to call a process_chunk function after. This should enable a caller to parallelize calls to sp::try_find_legacy_enotes_in_tx, and then process any few received enotes and identify spends in a much quicker second pass (which for a multi-wallet view-only scanner, can also be parallelized easily).

Compare this to wallet2's process_parsed_blocks which does everything in one pass, and handles threading internally in a fine-grained fashion.

The Seraphis lib is a clear upgrade. I think it's also worth noting I had an enjoyable experience using and reading the Seraphis lib code to write this code, which is something I generally can't say the same for when I try to do something with or in wallet2. I think the most bothersome thing to me was type conversion, which may have some room for improvement.

As far as exposing a C ABI, perhaps some external lib could be written that does something like what Cake Wallet does, exposing functions from within the Seraphis lib via a C wrapper, and includes instructions/examples on how to build the code for use with various languages? Perhaps there is a middle ground on this front compared to rewriting huge sections of the code in C?

Bonus: also check out the identify_receives function I implemented that shows how a view-only scanner could utilize the lib.