[Version: 3.8.0] DELETE request on Attachments causes persistent-queue to exhaust S3 connection pool, breaking subsequent requests

[PenjinakMeow]

Describe the bug
When I send a DELETE request to remove an attachment from an entity with Composition of many Attachments, the deletion itself works correctly, the attachment is removed from both the database and S3.

However, after the HTTP response completes (Finished is logged), the persistent-queue keeps calling retrieveClient() approximately 9 more times in the background.

This exhausts the generic-pool connection pool with unreleased S3 connections until it times out with ResourceRequest timed out. The main issue is that every subsequent request to the same service will fail with 500 Internal Server Error until the server is manually restarted. Below is the log for the issue.

[attachments] - [deleteAttachmentsWithKeys] Emitting DeleteAttachment for: 724900da-1159-4a92-b9c8-7f6a968f4c30
[attachments] - [deleteAttachmentsWithKeys] Emitted DeleteAttachment for: 724900da-1159-4a92-b9c8-7f6a968f4c30
[attachments] - [deleteAttachmentsWithKeys] Finished
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Fetching object store credentials for tenant shared. Using tenant-specific object store.
[attachments] - Creating S3 client {
  tenantID: 'shared',
  region: 'ap-southeast-1',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - s3 client has been created successfully {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx',
  region: 'ap-southeast-1'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[attachments] - Retrieving S3 client for { tenantID: 'shared' }
[attachments] - Using cached S3 client {
  tenantID: 'shared',
  bucket: 'hcp-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}
[persistent-queue] [ERROR] attachments: Emit failed: TimeoutError: ResourceRequest timed out
    at ResourceRequest._fireTimeout (/home/user/projects/expense-management/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
    at Timeout.bound (/home/user/projects/expense-management/node_modules/generic-pool/lib/ResourceRequest.js:8:15)
    at listOnTimeout (node:internal/timers:594:17)
    at process.processTimers (node:internal/timers:529:7)
[odata] [INFO] DELETE /odata/v4/cash-advance/Requests(ID=c1d1a79e-364d-4d01-bc60-d1910ce97624)/attachments(ID=88fbbce4-36b9-4764-ba18-d22f03279cc8) 
[error] [ERROR] 500 - TimeoutError: ResourceRequest timed out
    at ResourceRequest._fireTimeout (/home/user/projects/expense-management/node_modules/generic-pool/lib/ResourceRequest.js:62:17)
    at Timeout.bound (/home/user/projects/expense-management/node_modules/generic-pool/lib/ResourceRequest.js:8:15) {
  code: ''
}

To Reproduce

1. Set up a CAP entity with Composition of many Attachments (see entity definition below)

    requestNumber         : String     @assert.unique  @plugin.numberrange.rangeid: 'REQUESTID';
    employee              : Employee;
    title                 : String;
    advancedAmount        : Double;
    advancedCurrency      : S4Currency;
    defaultCurrency       : S4Currency @readonly;
    defaultCurrencyAmount : Double;
    country               : Country;
    location              : String;
    exchangeRate          : Double; 
    purpose               : String;
    requestDate           : TravelDate; 
    requestType           : RequestType;
    status                : Status     @readonly;
    taNumber              : String; 
    workflowID            : String;
    sapDocumentNumber     : String;
    taskID                : String;
    requestOnBehalf       : Boolean;
    expenseClaim          : Association to one Reports
                                on expenseClaim.request = $self;
    attachments           : Composition of many Attachments;
};

2. Upload an attachment to the entity
3. Send a single DELETE request to remove the attachment:
4. DELETE /odata/v4/cash-advance/Requests(ID=)/attachments(ID=)
5. Observe the logs — retrieveClient() is called ~10 times after Finished is logged
6. Send any subsequent GET/POST request to the same service
7. Observe 500 Internal Server Error: ResourceRequest timed out

Expected behavior
A single DELETE request should trigger retrieveClient() exactly once to delete the object from S3, and the connection should be released back to the pool cleanly. Subsequent requests to the service should not be affected.

[github-actions]

👋 Hello @PenjinakMeow, thank you for submitting this issue. Our team is reviewing your report and will follow up with you as soon as possible.

[PenjinakMeow]

Hi. Are there any updates on this issue? Thanks in advance

[KoblerS]

Hi @PenjinakMeow, could you please provide us a minimal reproducible example as a repository?