RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer by rameeshm · Pull Request #847 · apache/ranger

rameeshm · 2026-02-11T19:32:00Z

What changes were proposed in this pull request?

Audit Server should have service which has kafka as a queue mechanism and have Kafka Producer to relay and store audits in Kafka Topic / Partition.
Audit Server should have Solr consumer running as a separate service to write audit into SOLR Index as a destination
Audit Server should have HDFS consumer running as a separate service to write audit into HDFS.
All the above mentioned service should be available and test in docker container.

How was this patch tested?

Testing done in Docker running the containers of Ranger Audit Server with Ranger and HDFS and HIVE for policy and audit creation. Documentation on the JIRA has the testing details as well
https://docs.google.com/document/d/1a6ZZJyD6Mr7tIV1A7YknobjgSitcYzS9bDJ032MvjMc/edit?tab=t.0#heading=h.iitwqftxhxfq

…nsumer

Copilot

Pull request overview

This PR introduces a comprehensive Ranger Audit Server architecture that decouples audit collection from Ranger Admin by implementing:

A centralized Audit Server that receives audits from plugins via REST API and produces them to Kafka
Separate consumer services for SOLR and HDFS that consume from Kafka and write to their respective destinations
Shared common infrastructure for Kafka consumer management with support for consumer groups and rebalancing

Changes:

New audit server infrastructure with 3 microservices (audit-server, consumer-solr, consumer-hdfs) and shared common module
Integration with existing Ranger plugins (HDFS, Hive) to support HTTP-based audit destination
Docker support for testing the complete audit pipeline with Kerberos authentication
Thread-safe audit handling with concurrent counter management

Reviewed changes

Copilot reviewed 110 out of 111 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ranger-audit-server/scripts/*	Shell scripts for starting/stopping audit services
ranger-audit-server/ranger-audit-server-service/*	Core audit server service with REST API and Kafka producer
ranger-audit-server/ranger-audit-consumer-solr/*	SOLR consumer service with Kafka-to-SOLR pipeline
ranger-audit-server/ranger-audit-consumer-hdfs/*	HDFS consumer service with Kafka-to-HDFS pipeline
ranger-audit-server/ranger-audit-common/*	Shared utilities, base classes, and consumer registry
agents-audit/dest-auditserver/*	New audit destination for plugins to send to audit server
agents-common/src/main/java/org/apache/ranger/plugin/audit/*	Enhanced audit handler with service type in additional info
agents-audit/core/src/main/java/org/apache/ranger/audit/provider/*	Thread-safe counters using AtomicLong
dev-support/ranger-docker/*	Docker compose and Dockerfiles for audit server services
pom.xml, distro/*	Build configuration and assembly descriptors

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…nsumer - fix failing testing

…nsumer - fix pmd issue

…onsumer - Fix audit commit failure propagation and recovery in the consumers

Copilot

Pull request overview

Copilot reviewed 110 out of 111 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…nsumer - audit server partition management enhancement

mneethiraj

@rameeshm - sending comments of the portions I reviewed so far. Will continue reviewing.

...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java

agents-audit/core/src/main/java/org/apache/ranger/audit/provider/BaseAuditHandler.java

...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java

…nsumer - Fix review comments

…nsumer - PojoMappingFeature for AuditEvent Object for serialization

…onsumer - Fix review comments set #2

…nsumer - Audit Batch processing and failure reprocessing improvement

...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java

…onsumer - Fix duplicate dependency error in the pom for sl4j

…nsumer - Fix ubuntu audit ranger module war file creation failure

mneethiraj · 2026-02-28T08:35:23Z

...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java

+            return ret;
+        }
+
+        if (!serviceName.equals(authenticatedUser)) {


Requiring serviceName same as the authenticatedUser doesn't seem right. I guess the intention is that the caller is the identity of the service account for the specified serviceName. This identity should be obtained from the service configuration retrieved from Ranger admin - for example configurations service.admin.users. If necessary, we can introduce a new configuration service.account.user for this purpose.

Copilot

Pull request overview

Copilot reviewed 117 out of 118 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (7)

ranger-audit-server/ranger-audit-server-service/src/main/java/javax/ws/rs/core/NoContentException.java:1

Defining a class under the javax.ws.rs.* namespace inside the project is very risky: it can conflict with the JAX-RS API/implementation provided by the container or dependency set, and can lead to classloading/linkage issues that are hard to diagnose. Prefer fixing this by adding/aligning the correct JAX-RS/Jersey dependency that provides javax.ws.rs.core.NoContentException, or by relocating/shading any workaround class into a project-owned package and updating the reference point that needs it.
ranger-audit-server/ranger-audit-server-service/src/main/java/javax/ws/rs/core/NoContentException.java:1
Defining a class under the javax.ws.rs.* namespace inside the project is very risky: it can conflict with the JAX-RS API/implementation provided by the container or dependency set, and can lead to classloading/linkage issues that are hard to diagnose. Prefer fixing this by adding/aligning the correct JAX-RS/Jersey dependency that provides javax.ws.rs.core.NoContentException, or by relocating/shading any workaround class into a project-owned package and updating the reference point that needs it.
hdfs-agent/scripts/install.properties:1
The HDFS agent install properties set XAAUDIT.AUDITSERVER.FILE_SPOOL_DIR to a Hive log path (/var/log/hive/...). This looks like a copy/paste error and will cause HDFS plugin audit spooling to go to the wrong directory. Update it to an HDFS-appropriate directory (consistent with other HDFS audit spool paths in this repo).
ranger-audit-server/ranger-audit-server-service/src/main/webapp/WEB-INF/security-applicationContext.xml:1
CSRF is explicitly disabled while sessions are forced (create-session=\"always\"). If this service is intended to be a REST-only API (JWT/delegation-token), prefer making it stateless (no forced sessions) and limiting CSRF disabling to only the endpoints/methods that require it; otherwise, leaving CSRF off while maintaining session state increases risk for browser-based clients. Consider switching to stateless session creation and/or enabling CSRF with appropriate exemptions for non-browser clients.
ranger-audit-server/ranger-audit-server-service/src/main/webapp/WEB-INF/security-applicationContext.xml:1
CSRF is explicitly disabled while sessions are forced (create-session=\"always\"). If this service is intended to be a REST-only API (JWT/delegation-token), prefer making it stateless (no forced sessions) and limiting CSRF disabling to only the endpoints/methods that require it; otherwise, leaving CSRF off while maintaining session state increases risk for browser-based clients. Consider switching to stateless session creation and/or enabling CSRF with appropriate exemptions for non-browser clients.
ranger-audit-server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/producer/kafka/AuditRecoveryManager.java:1
RecoveryStats.toString() starts with RecoveryStats{ but never appends a closing }. This makes logs harder to read/parse. Append a closing brace before returning the string.
ranger-audit-server/ranger-audit-consumer-solr/src/main/java/org/apache/ranger/audit/rest/HealthCheckREST.java:1
A new ObjectMapper is created for every health check request. ObjectMapper is designed to be reused and can be expensive to construct. Prefer using a static/singleton ObjectMapper (or injecting one via Spring) to reduce allocation overhead.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

agents-common/src/test/java/org/apache/ranger/plugin/audit/TestRangerDefaultAuditHandler.java

agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java

agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java

Copilot · 2026-03-05T07:44:06Z

dev-support/ranger-docker/Dockerfile.ranger-audit-server

+COPY ./scripts/audit-server/service-check-functions.sh    ${RANGER_SCRIPTS}/
+COPY ./scripts/audit-server/ranger-audit-server.sh        ${RANGER_SCRIPTS}/


The Dockerfile copies scripts to ${RANGER_SCRIPTS}/ but the ENTRYPOINT uses a hard-coded /home/ranger/scripts/... path. If ${RANGER_SCRIPTS} is not exactly /home/ranger/scripts, the container will fail to start. Use a consistent path (either copy to /home/ranger/scripts explicitly, or set ENTRYPOINT to ${RANGER_SCRIPTS}/ranger-audit-server.sh).

Copilot · 2026-03-05T07:44:07Z

dev-support/ranger-docker/Dockerfile.ranger-audit-server

+
+# Start the audit server using the custom startup script
+WORKDIR /opt/ranger-audit-server
+ENTRYPOINT ["/home/ranger/scripts/ranger-audit-server.sh"]


The Dockerfile copies scripts to ${RANGER_SCRIPTS}/ but the ENTRYPOINT uses a hard-coded /home/ranger/scripts/... path. If ${RANGER_SCRIPTS} is not exactly /home/ranger/scripts, the container will fail to start. Use a consistent path (either copy to /home/ranger/scripts explicitly, or set ENTRYPOINT to ${RANGER_SCRIPTS}/ranger-audit-server.sh).

Suggested change

ENTRYPOINT ["/home/ranger/scripts/ranger-audit-server.sh"]

ENTRYPOINT ${RANGER_SCRIPTS}/ranger-audit-server.sh

agents-audit/core/src/main/java/org/apache/ranger/audit/model/AuthzAuditEvent.java

agents-audit/core/src/main/java/org/apache/ranger/audit/provider/AuditHandler.java

agents-audit/core/pom.xml

mneethiraj · 2026-03-05T07:56:43Z

...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java

+        }
+    }
+
+    private void preAuthenticateKerberos() {


preAuthenticateKerberos() => initKerberos()

mneethiraj · 2026-03-05T08:00:14Z

...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java

+
+        // Add serviceName to query parameters
+        if (StringUtils.isNotEmpty(serviceType)) {
+            queryParams.put(QUERY_PARAM_SERVICE_NAME, serviceType);


Parameter QUERY_PARAM_SERVICE_NAME is set to serviceType? Perhaps the parameter should be renamed to QUERY_PARAM_SERVICE_TYPE?

mneethiraj · 2026-03-05T08:05:58Z

...itserver/src/main/java/org/apache/ranger/audit/destination/RangerAuditServerDestination.java

+    public static final String PROP_AUDITSERVER_SSL_CONFIG_FILE              = "xasecure.audit.destination.auditserver.ssl.config.file";
+    public static final String PROP_AUDITSERVER_MAX_RETRY_ATTEMPTS           = "xasecure.audit.destination.auditserver.max.retry.attempts";
+    public static final String PROP_AUDITSERVER_RETRY_INTERVAL_MS            = "xasecure.audit.destination.auditserver.retry.interval.ms";
+    public static final String PROP_SERVICE_TYPE                             = "ranger.plugin.audit.service.type";


Why are serviceType and appId needed for audit destination? How about using existing fields in AuthzAuditEvent - repoType (serviceType), repositoryName (serviceName), agentType (appId) instead, if necessary?

It will help avoid additional configurations that need to be setup in plugin.

mneethiraj · 2026-03-05T08:11:13Z

agents-audit/dest-auditserver/pom.xml

+        </dependency>
+        <dependency>
+            <groupId>com.google.code.gson</groupId>
+            <artifactId>gson</artifactId>


gson doesn't seem to be used. Please review and remove if unused.

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

54f64ef

…nsumer

rameeshm requested review from Copilot, dineshkumar-yadav, kumaab, mneethiraj, pradeepagrawal8184 and vperiasamy February 11, 2026 19:32

Copilot started reviewing on behalf of rameeshm February 11, 2026 19:36 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

Ramesh Mani added 3 commits February 11, 2026 13:42

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

ce3542b

…nsumer - fix failing testing

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

0004b63

…nsumer - fix pmd issue

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit c…

e6bdea2

…onsumer - Fix audit commit failure propagation and recovery in the consumers

rameeshm requested review from Copilot and vikaskr22 February 18, 2026 20:44

Copilot started reviewing on behalf of rameeshm February 18, 2026 21:05 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

bbbb191

…nsumer - audit server partition management enhancement

mneethiraj reviewed Feb 20, 2026

View reviewed changes

Ramesh Mani added 4 commits February 23, 2026 00:20

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

cb086a0

…nsumer - Fix review comments

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

878b4ab

…nsumer - PojoMappingFeature for AuditEvent Object for serialization

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit c…

24cebf3

…onsumer - Fix review comments set #2

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

87d9c0c

…nsumer - Audit Batch processing and failure reprocessing improvement

rameeshm commented Feb 25, 2026

View reviewed changes

...server/ranger-audit-server-service/src/main/java/org/apache/ranger/audit/rest/AuditREST.java Outdated Show resolved Hide resolved

Ramesh Mani added 2 commits February 25, 2026 16:36

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit c…

63c9405

…onsumer - Fix duplicate dependency error in the pom for sl4j

RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit co…

4536899

…nsumer - Fix ubuntu audit ranger module war file creation failure

mneethiraj reviewed Feb 28, 2026

View reviewed changes

mneethiraj requested a review from Copilot March 5, 2026 07:39

Copilot AI reviewed Mar 5, 2026

View reviewed changes

mneethiraj reviewed Mar 5, 2026

View reviewed changes

		COPY ./scripts/audit-server/service-check-functions.sh ${RANGER_SCRIPTS}/
		COPY ./scripts/audit-server/ranger-audit-server.sh ${RANGER_SCRIPTS}/

	ENTRYPOINT ["/home/ranger/scripts/ranger-audit-server.sh"]
	ENTRYPOINT ${RANGER_SCRIPTS}/ranger-audit-server.sh

Conversation

rameeshm commented Feb 11, 2026

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

mneethiraj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mneethiraj Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mneethiraj Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mneethiraj Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

mneethiraj Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants