Name endpoints pagination

[nodech]

Wallet pagination for auctions

List of auctions, bids and reveals suffers from the same issues as tx history did (#888). This is document describes the potential API and reindexing necessary to get the similar features as the tx history. The indexes are categorized two ways: time index and count index. Count index refers to the ordered list of items, where adding new entries from new blocks does not change the ordering of the previous ones. Wallet TX pagination uses Block height + tx index in the block - for this, and time index is references into this.
Names have four separate endpoints and entries in the db: namestate, auction, bid and reveal. The indexing and pagination should be consistent across these three. Different from txs, names are only indexed for the confirmed txs. All these have two separate indexes for the owned and non-owned. Names and auctions currently are the same, but should be separate.

Time index note: Because all entries are only indexed by block, their time will be same as the block median time. So instead of having separate time indexes, we can introduce new index for median time -> block height, which can be used for querying by time. (Note: Confirmed tx index for time could also be removed and use time -> height index)

TX Pagination uses tx hashes as the cursor, unfortunately same can't be done with the auctions. Same name can appear several, after names expire - which are essentially different auctions. We can instead use block height + tx index (returned via API/Seek) in order to iterate over auctions/bids/reveals. NOTE: TXs use block height + index internally.
Sorted index for the auctions can be fully expressed by combination of height + tx index (as names themselves are not unique and have multiple auctions, we can't use them as cursor). We will go into details per item below.

NOTE: layout keys not chosen for this document, it must be done on implementation step. Listed layouts will use descriptive names instead of one symbol prefixes.

General block time -> height index

• layout.heightByTime[time] -> height - Uses monotonic time.

General considerations

• Wherever height + tx-index occurs, it can be aliased to txhash and use TX pagination entries to look up height and index (TXCount). It makes no sense for some: E.g. auctions. But this could be useful for bids/reveals.

Names (Latest namestate)

Names are the latest state that wallet knows from the chain. This is not an auction, there can be multiple auctions per name. NOTE: This endpoint wont have account to it, as namestate is more general state.

Indexes:

• layout.namestateUniqueIndex -> number.
• layout.namestateByIndex[uniqueNameIndex++] -> namestate
  • uniqueIndex is there to make sure, next/prev page does not get shifted when new name is indexed.
    This is used to iterate over all names in consistent manner.
• layout.namestateOwnedByIndex[uniqueNameIndex] -> dummy
  • This allows us to iteretate over owned names separately given unique name index and resolve the name
    from the namestateByIndex entries.
• layout.namestateIndexByName[nameHash] -> uniqueNameIndex
  • Given a name cursor, we can internally resolve the index, this is used by owned/non-owned. Also ensures we don't have name with two indexes.

HTTP API:

• GET /wallet/:id/name - get latest namestates, but with new parameters.

  • params
    • reverse - boolean - should we reverse the list.
    • limit - number - number of namestates to list at once.
    • after - string - list of namestates after the name.
    • owned - boolean - list of namestates that are owned.
  • Uses layout.namestateByIndex for iteration.
    • Uses layout.namestateOwnedByIndex if owned is set.
  • Uses layout.namestateIndexByName[nameHash] to get from cursor to internal iteration.
    This endpoint does not have time param, as this refers to the latest state instead of auction.

• GET /wallet/:id/name/:name - same as it was.

  • Uses layout.namestateIndexByName[nameHash] -> layout.namestateByIndex[index]

Auctions

Auctions are almost same as names, but they can happen several times if the name expires. They can be uniquely identified by the block height + name, but tx-index is there to ensure the ordering follows blockchains ordering in a block.

Indexes:

• layout.auctionsByCount[auction-height][tx-index][nameHash] -> namestate
  • Can have multiple names under same height.
  • This can be used to list all auctions by chronological order.
• layout.auctionsByName[nameHash][auction-height][tx-index] -> dummy (uses auctionsByCount to resolve)
  • or layout.auctionsByName[nameHash][auction-height] -> tx-index (tx-index does not matter for sorting)
  • Can have multiple heights under same name.
  • tx-index is only used for recovering auctionsByCount entry.

Account indexes:

• layout.auctionsByCountAccount[account][height][tx-index][nameHash] -> dummy
• layout.auctionsByNameAccount[account][nameHash][height][tx-index] -> dummy
  • or layout.auctionsByNameAccount[account][nameHash][height] -> tx-index (tx-index does not matter for sorting)

HTTP API:

• GET /wallet/:id/auction - get all wallet auctions. (Must no longer include bids/reveals)

  • params:
    • account - optional string - instead of whole wallet, do specific account.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list auctions after timestamp(or before reverse)
      • Uses layout.heightByTime[time]
    • after - height:txindex - cursor is little bit tricky here. Cursor alternatives:
      • height <<< ceil(log2(MAX_TX_IN_BLOCK)) | tx-index - gives a single number.
      • name:height - this has advantage that the namestate json contains these two.
      • (Or afterName, afterHeight)
  • Uses layout.auctionsByCount[height][index] for iteration (with or w/o after).
    • Uses layout.auctionsByCountAccount[height][index] if account is set.

• GET /wallet/:id/auction/:name - Get all wallet auctions for the name. (Must no longer include bids/reveals)

  • params
    • account - optional string - instead of whole wallet, do specific account lookup.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list auctions after timestamp(or before reverse)
      • Uses layout.heightByTime[time]
    • after - height.
  • Uses layout.auctionsByName[name][height] to iterate over name auctions (with or w/o after).
    • Uses layout.auctionsByNameAccount[account][nameHash][height] if account is set.

• GET /wallet/:id/auction/:name/:height - Get specific auction.

  • params
    • account - optional string - make sure we check specific account only.
  • Uses layout.auctionsByName[nameHash][height] to grab specific auction.
    • Uses layout.auctionsByNameAccount[account][nameHash][height] if account is set.

This also enables Names endpoint:

• GET /wallet/:id/name/:name/:height - to get namestate for specific auction, not latest. Similar to historical proofs, but namestates.
  • Uses layout.auctionsByName[nameHash][height] to get indexes.

Bids

Bids are different from auctions. Block and TX can have multiple bids for the wallet, so if we want sorted set we need to include all this. Another important note: Not all bids for the same name are part of the same auction, so they need to be separated as well.

Indexes:

• layout.bidByOutput[height][tx-index][out-index] -> BlindBid
  • This does not need auction height, it can be extracted from the BlindBid, sorting is good enough with height.
• layout.bidByOutputOwned[height][tx-index][out-index] -> dummy
• layout.bidByName[nameHash][height][tx-index][out-index] -> dummy
• layout.bidByNameOwned[nameHash][height][tx-index][out-index] -> dummy

Account indexes:

• layout.bidByOutputAccount[account][height][tx-index][out-index] -> BlindBid
  • This does not need auction height, it can be extracted from the BlindBid, sorting is good enough with height.
• layout.bidByOutputOwnedAccount[account][height][tx-index][out-index] -> dummy
• layout.bidByNameAccount[account][nameHash][height][tx-index][out-index] -> dummy
• layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummy

HTTP API:

• GET /wallet/:id/bid - get all wallet bids.

  • params
    • account filter by account.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list bids after timestamp
      • Uses layout.heightByTime[time]
    • after - This could be built using height + tx-index + out-index.
      • e.g. ?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.
    • owned - Filter the bids that are ours only.
  • Uses layout.bidByOutput[height][tx-index][out-index] index for iteration.
    • Uses layout.bidByOutputAccount[account][height][tx-index][out-index] if account is set.
    • Uses layout.bidByOutputOwned[height][tx-index][out-index] if owned is set.
    • Uses layout.bidByOutputOwnedAccount[account][height][tx-index][out-index] if account and owned is set.

• GET /wallet/:id/bid/:name - get all bids specific to name. (Include all auctions)

  • params
    • account filter by account.
      • Uses Account Indexes instead of normal indexes for all operations.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list bids after timestamp
      • Uses layout.heightByTime[time]
    • after - This is same as all bids: height + tx-index + out-index
    • owned - Filter the bids that are ours only.
  • Uses layout.bidByName[nameHash][height][tx-index][out-index] index.
    • Uses layout.bidByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
    • Uses layout.bidByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
    • Uses layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.

This also enables Auctions endpoint:

• GET /wallet/:id/auction/:name/:height/bid - get all bids specific to single auction.
  • params
    • account filter by account.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list bids after timestamp
      • Uses layout.heightByTime[time]
    • after - This same as well: height + tx-index + out-index
    • owned - Filter the bids that are ours only.
  • Uses layout.auctionsByName[name][height] to get the start height, end height can be grabbed with treeInterval + biddingPeriod
    • Uses layout.bidByName[nameHash][height][tx-index][out-index]
    • Uses layout.bidByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
    • Uses layout.bidByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
    • Uses layout.bidByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.

Reveals

Reveals are similar to bids. There can be several reveals in a single block and in a single tx.

Indexes:

• layout.revealByOutput[height][tx-index][out-index] -> BidReveal
  • This does not need auction height, it can be extracted from the BidReveal, sorting is good enough with height.
• layout.revealByOutputOwned[height][tx-index][out-index] -> dummy
• layout.revealByName[nameHash][height][tx-index][out-index] -> dummy
• layout.revealByNameOwned[nameHash][height][tx-index][out-index] -> dummy

Account indexes:

• layout.revealByOutputAccount[account][height][tx-index][out-index] -> BidReveal
  • This does not need auction height, it can be extracted from the BidReveal, sorting is good enough with height.
• layout.revealByOutputOwnedAccount[account][height][tx-index][out-index] -> dummy
• layout.revealByNameAccount[account][nameHash][height][tx-index][out-index] -> dummy
• layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] -> dummy

HTTP API:

• GET /wallet/:id/reveal - get all wallet reveals.

  • params
    • account filter by account.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list bids after timestamp
      • Uses layout.heightByTime[time]
    • after - This could be built using height + tx-index + out-index.
      • e.g. ?height=200100&txIndex=10&outIndex=5, unfortunately, similar to Auctions API, it's not easy to specify these manually and only more sense when used on iteration, built from last entry.
    • owned - Filter the reveals that are ours only.
  • Uses layout.revealByOutput[height][tx-index][out-index] index for iteration.
    • Uses layout.revealByOutputAccount[account][height][tx-index][out-index] if account is set.
    • Uses layout.revealByOutputOwned[height][tx-index][out-index] if owned is set.
    • Uses layout.revealByOutputOwnedAccount[account][height][tx-index][out-index] if account and owned is set.

• GET /wallet/:id/reveal/:name - get all reveals specific to name. (Include all auctions)

  • params
    • account filter by account.
      • Uses Account Indexes instead of normal indexes for all operations.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list reveals after timestamp
      • Uses layout.heightByTime[time]
    • after - This is same as all reveals: height + tx-index + out-index
    • owned - Filter the reveals that are ours only.
  • Uses layout.revealByName[nameHash][height][tx-index][out-index] index.
    • Uses layout.revealByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
    • Uses layout.revealByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
    • Uses layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.

This also enables Auctions endpoint:

• GET /wallet/:id/auction/:name/:height/reveal - get all reveals specific to single auction.
  • params
    • account filter by account.
    • reverse - boolean - should we reverse the list.
    • limit - number - number of auctions to list at once.
    • time - number - list reveals after timestamp
      • Uses layout.heightByTime[time]
    • after - This same as well: height + tx-index + out-index
    • owned - Filter the reveals that are ours only.
  • Uses layout.auctionsByName[name][height] to get the start height, end height can be grabbed with treeInterval + revealdingPeriod
    • Uses layout.revealByName[nameHash][height][tx-index][out-index]
    • Uses layout.revealByNameAccount[account][nameHash][height][tx-index][out-index] if account is set.
    • Uses layout.revealByNameOwned[nameHash][height][tx-index][out-index] if owned is set.
    • Uses layout.revealByNameOwnedAccount[account][nameHash][height][tx-index][out-index] if owned and account is set.