Feature/ingest v2 by ag-ramachandran · Pull Request #439 · Azure/azure-kusto-java

ag-ramachandran · 2025-09-09T05:59:32Z

Initial work on the IngestV2 API

Copilot

Pull Request Overview

This PR introduces the initial work on the IngestV2 API, implementing a new Kotlin-based ingestion service alongside the existing Java implementation. The major change is upgrading the project from Java 8 to Java 11 and updating various dependencies to support the new ingest-v2 module.

Project upgraded from Java 8 to Java 11 with dependency version updates
New ingest-v2 module written in Kotlin with modern async capabilities
Centralized dependency management moved to root POM for consistency

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pom.xml	Upgraded Java version, updated dependencies, added ingest-v2 module, centralized dependency management
ingest/pom.xml	Removed duplicate dependency management, cleaned up version specifications
data/pom.xml	Removed duplicate dependency management, cleaned up version specifications
ingest-v2/pom.xml	Complete new module configuration with Kotlin, Ktor, and OpenAPI generation
ingest-v2/src/main/resources/openapi.yaml	OpenAPI specification for the new ingest REST API
ingest-v2/src/main/kotlin/...	Core Kotlin implementation files for authentication, retry policies, data sources, and API clients
ingest-v2/src/test/kotlin/...	Unit tests for retry policies and configuration API

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

...main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/auth/AzCliTokenCredentialsProvider.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/BlobSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/LocalSource.kt

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/utils/PathUtils.kt

github-actions · 2025-09-09T06:00:40Z

Test Results

530 tests ±0 521 ✅ ±0 9m 50s ⏱️ + 6m 47s
31 suites ±0 9 💤 ±0
31 files ±0 0 ❌ ±0

Results for commit d0d2046. ± Comparison against base commit 85e4eec.

♻️ This comment has been updated with latest results.

* added private link access context * Some refactor and add changes for ONE_LAKE_UPLOAD test * Fix uploader using DFS * Add additional test context for failures * Add error details * Add some info on ClientDetails --------- Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>

* * Refactor to add GZIP Compression * Add a simple streaming sample * Fix issue with compression on Streaming data ingest * Fix copilot review comments * Fix imports * Add assertions --------- Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>

* Remove lints

* * Make signatures consistent for all sources

* Reformat code * Add alias for tests

* Added Unit tests for coverage * Rebase changes base branch * Reformat tests --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

…nd unit tests

* * Fix review comments - Make the IngestRequestProperties optional * * Fix review comments - Make sure some of the validations like passing IngestionReference and mapping cannot be passed together * * Fix review comments - Make sure some of the validations like passing IngestionReference and mapping cannot be passed together * Add tests for QueuedIngest with inline mapping * * Fix breaking changes to IngestRequestProperties * * Add tests for IngestRequestProperties being optional * * Remove baseName parameter * * Fix review comments * * Fix review comments * * Fix review comments * * Fix review comments

* added code for wellknown kusto endpoints * addressed review comments

* fixed review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

* * Fix method signature for uploads * * Minor edit to ConfigurationCache to determine refresh interval logic * * Additional tests for preferred upload combinations * * Reformat tests * Add tests for duration

Copilot

Pull request overview

Copilot reviewed 110 out of 116 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

AsafMah · 2026-01-28T12:10:13Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/auth/endpoints/FastSuffixMatcher.kt

@@ -0,0 +1,119 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+package com.microsoft.azure.kusto.ingest.v2.auth.endpoints


I don't like that we have to duplicate these entire classes.

I know we discussed before dependencies on the data package, but at this point we need to reconsider if it leaves us with this much duplicate code.

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

AsafMah · 2026-01-28T12:14:00Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

+
+    protected abstract fun self(): T
+
+    fun withAuthentication(credential: TokenCredential): T {


In .net, we have withNoAuthentication, here you don't, and instead throw an error if it's not called.

Either it should be part of the main ctor, or should work like c#

AsafMah · 2026-01-28T12:15:53Z

...t-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/builders/BaseIngestClientBuilder.kt

+    }
+
+    companion object {
+        protected fun normalizeAndCheckEngineUrl(clusterUrl: String): String {


These methods are wrong.

See the java getIngestionEndpoint, we have more rules there for special urls. It's also more efficient without constantly building regexes.

Another reason I think to depend on kusto-data.

samples/src/main/java/ingestv2/QueuedIngestV2.java

AsafMah · 2026-01-28T12:23:09Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/client/QueuedIngestClient.kt

+     * sent directly to the Data Management service.
+     */
+    @JvmName("ingestAsync")
+    fun ingestAsyncJava(


So all of these methods are not part of the interface?

So if someone uses the interface the can't use it in java?

AsafMah · 2026-01-28T12:34:02Z

ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/uploader/ContainerUploaderBase.kt

+     * Atomic counter for round-robin container selection. Increments on each
+     * upload to distribute load evenly across containers.
+     */
+    private val containerIndexCounter = AtomicInteger(0)


Maybe I'm missing something, but that's not what we want -
This has a counter per uploader.

Here, if uploader A and uploader B (sharing the same configurationcache) upload, they will both use storage 0 first.

The state should be a part of the specific list inside configuration cache - so uploader A will use storage 0, and uploader B will use storage 1. See how we do it in c#

* addressed review comments * addressed review comments * simplified container selection logic and added tests

* added CachingTokenCredential for concurrent azcli token handling * simplified token generation and access

* Apply formatter and add documentation * * Reformat files

ag-ramachandran requested review from AsafMah, Copilot and tanmaya-panda1 September 9, 2025 05:59

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 05:59 — with GitHub Actions Error

Copilot AI reviewed Sep 9, 2025

View reviewed changes

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 10:10 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 9, 2025 16:39 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Failure

ag-ramachandran had a problem deploying to build September 10, 2025 01:24 — with GitHub Actions Error

ag-ramachandran temporarily deployed to build September 10, 2025 01:57 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:15 — with GitHub Actions Inactive

ag-ramachandran temporarily deployed to build September 10, 2025 12:20 — with GitHub Actions Inactive

AsafMah marked this pull request as ready for review September 18, 2025 09:12

tanmaya-panda1 and others added 22 commits January 8, 2026 08:36

disabled serialized execution

ba3f817

optimized QueuedIngestionClientTests execution

ab64a1b

* Remove test for missing blob

4a17c49

* Remove lints

* Rebase POM changes

307ff6a

* Rebase POM changes

e85af02

* Rebase POM changes

1d0a6fa

* Add JACOCO for coverage

3a31d4d

Feature/fix method signature orders (#452)

4931ac0

* * Make signatures consistent for all sources

* Rebase changes with master

bca955e

* Reformat code * Add alias for tests

Feature/add junits (#451)

d6a66f7

* Added Unit tests for coverage * Rebase changes base branch * Reformat tests --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

* Remove unused test assertion

4ebf781

* Remove JVMOverloads and add log supression for bytebuddy in MOCKK a…

d3428ec

…nd unit tests

* Fix lint in tests

bdce366

* Minor changes to Streaming ingest tests (#454)

b9ccf14

added code for wellknown kusto endpoints (#456)

8a41876

* added code for wellknown kusto endpoints * addressed review comments

Users/tanmayapanda/address review comments (#458)

66de6e7

* fixed review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>

addressed review comments (#460)

4c3d8f6

Users/ramacg/refactor managed uploader (#455)

5f296db

* * Fix method signature for uploads * * Minor edit to ConfigurationCache to determine refresh interval logic * * Additional tests for preferred upload combinations * * Reformat tests * Add tests for duration

* Fix JDK version in POM (#461)

b1ec311

tanmaya-panda1 approved these changes Jan 21, 2026

View reviewed changes

Copilot AI reviewed Jan 21, 2026

View reviewed changes

added private link and removed tuple/Pair (#462)

f61924c

AsafMah requested changes Jan 28, 2026

View reviewed changes

tanmaya-panda1 and others added 4 commits February 18, 2026 18:28

addressed review comments (#463)

bad4866

* addressed review comments * addressed review comments * simplified container selection logic and added tests

fix test failing due to concurrent azcli token invocation (#464)

0e7c1fa

* added CachingTokenCredential for concurrent azcli token handling * simplified token generation and access

* Refactor and add tests for KustoBaseApiClient (#465)

7dbbf30

Users/ramacg/add doc and format code (#466)

d0d2046

* Apply formatter and add documentation * * Reformat files

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/ingest v2#439

Feature/ingest v2#439
ag-ramachandran wants to merge 54 commits intomasterfrom
feature/IngestV2

ag-ramachandran commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

AsafMah Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		protected abstract fun self(): T

		fun withAuthentication(credential: TokenCredential): T {

Conversation

ag-ramachandran commented Sep 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

AsafMah Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Sep 9, 2025 •

edited

Loading