Feature Request: Transactional Bulk Operations

[felixminom]

Hi Topaz team,

As we've been using Topaz more and more, we've started running into cases where having some kind of bulk operation support with transactional guarantees would be really helpful.

Right now, when you need to apply a group of related changes (objects, relationships, etc.) and ensure they either all succeed or all fail together, that logic has to live in the consumer. That ends up pushing a lot of complexity downstream, and the solutions tend to be non-native and different depending on who's integrating with Topaz.

We've also looked at using streams for this, but in practice, it hasn't been as reliable as we'd like for scenarios where consistency really matters.

So I wanted to ask: are there any plans to support transactional bulk operations natively?

If this is something the project would be interested in adding, we'd also be happy to collaborate on the design and implementation.

Thanks!

[gertd]

Hi @felixminom,

Can you please elaborate on exactly how you are using APIs? Especially the stream use case you're referring to? Which programming language are you using for this? Also, please describe your operational Topaz setup. Are you synchronizing between Topaz instances? Or are you using standalone edges? Etc.

API semantics:

• All Set and Delete operations are transacted
• In addition, (Set | Delete) Relation is idempotent, as the instance always equals the unique key, which can never change.
• The API supports optimistic concurrency (see below)
• If a Set or Delete call does not return an error, the underlying commit succeeded
• Relation and Object instance are 100% independent and can exist without each other (see below), there is no referential integrity as in a relational database. Types and allowed relations/permissions between types are validated against the manifest used before each operation hits the underlying store.

Are you checking the etag value of the response message?
Are you leveraging optimistic concurrency If-Match == etag?

Optimistic concurrency example using grpcurl (works the same for REST based clients)

grpcurl -insecure -d '{
  "objectType": "user",
  "objectId": "test0"
}' localhost:9292 aserto.directory.reader.v3.Reader.GetObjectreturns

Response

{
  "result": {
    "type": "user",
    "id": "test0",
    "display_name": "",
    "properties": {},
    "created_at": "2026-01-15T13:53:20.049165Z",
    "updated_at": "2026-01-15T13:53:20.049165Z",
    "etag": "2416901455385505062"
  },
  "relations": [],
  "page": {
    "next_token": ""
  }
}

Updating it with an incorrect etag

grpcurl -insecure -H "If-Match: 24169014553855050621" -d '{
  "object": {
    "type": "user",
    "id": "test0",
    "displayName": "test1"
  }
}' localhost:9292 aserto.directory.writer.v3.Writer.SetObject

Fails as the etag is not equal to the current one

ERROR:
  Code: FailedPrecondition
  Message: E20013 etag mismatch: for object with type [user] and id [test0]
  Details:
  1)    {
          "@type": "type.googleapis.com/google.rpc.ErrorInfo",
          "domain": "E20013",
          "metadata": {
            "aserto-http-statuscode": "412",
            "msg": "for object with type [user] and id [test0]"
          },
          "reason": "d7cbbcc4-f21e-11f0-94cc-e2aad50757b5"
        }

This is not the first request for adding transactional control, which was dismissed for the following reasons:

• object and relation instances are 100% autonmous instance, meaning one can have a Relation instance referring to subject type A id 1 and object type B id 2, but no actual Object instances. The relation object is always immutable, as the required values are the fully unique key and cannot be changed, only deleted. Objects can exist without being referenced by a relation; people use this as a general property bag to represent config or other things.
• Topaz and its commercial predecessors support(ed) multiple stores: key-value (BoltDB, NATS JetStream), relational (Postgres). Each underlying store needs to have full control, which makes it even harder to externalize that control if we wanted to.
• externalizing transactional control (being able to begin and end a transactional scope) creates a real (concurrency) risk, where incorrect usage impacts the availability of data, which is not desirable for a directory.
  • BoltDB is a single-writer store makes this risk even more real

Looking forward to learn about your use-case

[gertd]

One possible addition we have been discussing and prototyped for the v4 schema is a Batch request, which effectively is an array containing a oneof field that reuses the SetObjectRequest, DeleteObjectRequest, SetRelationRequest, and DeleteRelationRequest, returning an array of the corresponding response messages.

message BatchRequest {
  repeated BatchRequests requests = 1;
}

message BatchResponse {
  repeated BatchResponses responses = 1;
}

message BatchRequests {
  oneof msg {
    // set object request
    aserto.directory.writer.v4.SetObjectRequest set_object = 1;
    // delete object request
    aserto.directory.writer.v4.DeleteObjectRequest delete_object = 2;
    // set relation request
    aserto.directory.writer.v4.SetRelationRequest set_relation = 3;
    // delete relation request
    aserto.directory.writer.v4.DeleteRelationRequest delete_relation = 4;
  }
}

message BatchResponses {
  oneof msg {
    // set object request
    aserto.directory.writer.v4.SetObjectResponse set_object = 1;
    // delete object request
    aserto.directory.writer.v4.DeleteObjectResponse delete_object = 2;
    // set relation request
    aserto.directory.writer.v4.SetRelationResponse set_relation = 3;
    // delete relation request
    aserto.directory.writer.v4.DeleteRelationResponse delete_relation = 4;
  }
}

Usability wise this would look like:

grpcurl -insecure -d '{
  "requests": [
    {
      "setRelation": {
        "relation": {
          "objectType": "user",
          "objectId": "user-id-1",
          "relation": "identifier",
          "subjectType": "identity",
          "subjectId": "identity-id-1",
          "subjectRelation": ""
        }
      }
    },
    {
      "setObject": {
        "object": {
          "objectType": "user",
          "objectId": "user-id-1",
          "properties": {}
        }
      }
    },
    {
      "setObject": {
        "object": {
          "objectType": "identity",
          "objectId": "identity-id-1",
          "properties": {}
        }
      }
    }
  ]
}' localhost:9292 aserto.directory.writer.v4.Writer.Batch`

NOTES:

• The batch is an all-or-nothing; it either succeeds or fails.
• The request is gated by the max request size, 4Mb.
• The responses are the regular response of the corresponding request, located at the same ordinal as the request in the responses array.
• When we designed this, we believed the sweetspot use-case is the one above, with a single relation and two objects with a property bag (as without a property bag, the existence of an object instance does not add any value).

Interested in gathering input on this, as Batch was not added to the v4 directory schema.

[felixminom]

Hey @gertd — first of all, thank you for taking the time to respond to my issue, I really appreciate it.

I can add a bit more context around our use case and how we’re currently using the APIs.

Can you please elaborate on exactly how you are using APIs? Especially the stream use case you're referring to? Which programming language are you using for this?

We currently have a Kotlin service that acts as the single entry point for all Topaz-related operations. When we initially tried to implement batch deletion, we attempted to use the stream functionality. However, we ran into reliability issues: in practice, we often had to retry multiple times to ensure all relations were actually deleted.

Because of that, we ended up moving away from streams and instead issuing multiple individual calls to Topaz, deleting relations one by one. This approach has proven to be much more reliable for us, and as you mentioned, Delete operations being transactional and idempotent is a big advantage here.

Are you synchronizing between Topaz instances? Or are you using standalone edges?

We’re using a standalone instance — no synchronization between multiple Topaz instances.

Regarding the batch functionality you described, that would be a great addition to the ecosystem and is exactly the kind of behavior we’re trying to achieve. The goal is to support multiple types of messages executed in an all-or-nothing fashion. We’ve been attempting to recreate this behavior at the service level, but it’s more complex than it initially appears.

To give a concrete example:

• Our service receives a request to add and/or delete ~100 relations on the same "resource" .
• The service then attempts to execute these operations one by one.
• Now imagine that operation 51 fails due to a transient, retryable issue (for example, a network hiccup).

At that point, we’re left with two imperfect options:

• Retry the failed operations, which introduces eventual consistency into the system — something that isn’t acceptable for some of our use cases.

• Attempt to roll back by deleting the previously successful operations, which can also fail and potentially leave orphaned relations behind.

Having first-class batch support would eliminate these scenarios entirely by providing atomicity guarantees at the Topaz level. That’s the core reason we’re interested in this feature.

[ashwinrs]

Our Topaz Directory is backed by DynamoDB. We do think Batch kind of API could be helpful(group membership syncs) but don't necessarily see the need for it. The unary APIs are idempotent, so just retrying if it fails has worked for us.
That said, the interface defined in the previous comment would work fine with DynamoDB's transaction primitives.