feat: PostgreSQL-backed search/document storage as ElasticSearch alternative#700
feat: PostgreSQL-backed search/document storage as ElasticSearch alternative#700pandor4u wants to merge 2 commits intomoqui:masterfrom
Conversation
…rnative Add a new 'postgres' ElasticClient type that implements the full ElasticFacade.ElasticClient interface using PostgreSQL with JSONB document storage, tsvector full-text search, and GIN indexes. Key changes: - PostgresElasticClient: full ElasticClient implementation backed by PostgreSQL tables (moqui_search_index, moqui_logs, moqui_http_log) - ElasticQueryTranslator: translates ES Query DSL (bool, term, terms, range, nested, exists, match_all, query_string, ids) into parameterized PostgreSQL SQL with sanitized field names - PostgresSearchLogger: Log4j2 appender writing to PostgreSQL - SearchEntities.xml: entity definitions with JSONB, tsvector, GIN indexes - Security hardening: field name sanitization, parameterized queries, env-var credentials in Docker, TLS 1.2 minimum - Comprehensive test suite: 83 tests covering query translation, CRUD, bulk indexing, search, and SQL injection prevention Configuration: set elastic-client type="postgres" in Moqui XML conf. No external search engine dependency required.
Bug fixes for ElasticQueryTranslator: - Add tsqueryParams field separation to prevent parameter binding mismatch when bool queries combine query_string with term/range - Remove no-op AND→AND/OR→OR replacements in cleanLuceneQuery Bug fixes for PostgresElasticClient: - Fix bulk() delete pair skew: rewrite from fixed i+=2 stride to variable stride (delete consumes 1 item, others consume 2) - Add prefixIndexName() on action-spec _index in bulk() - Fix search() to use tq.tsqueryParams for SELECT score clause instead of tq.params (was misaligning parameter indexes) - Wrap CREATE EXTENSION pg_trgm in try-catch for managed DB envs Also adds AI instruction files (AGENTS.md, CLAUDE.md, GEMINI.md) and updates .gitignore for development artifacts.
|
Here is some feedback from GLM-5.1 PR #700 EvaluationFeature: Replaces ElasticSearch/OpenSearch with a PostgreSQL-backed search backend (JSONB + tsvector) via the existing What's in scope (appropriate for upstream)
What does NOT belong upstream
Key technical concerns
Overall assessmentThe architecture is solid — clean Strategy pattern, backward-compatible, good SQL injection prevention, proper parameterized queries. The main issue is that this PR mixes the legitimate framework feature with a lot of fork-specific baggage (3 identical AI instruction files, fork-specific gitignore entries, fork-specific docker compose). If you're evaluating this for upstream merge, those should be stripped out first, the integration test suite issue should be fixed, and the duplicated SQL should be consolidated. |
PostgreSQL-Backed Search & Document Storage
This PR adds a new
postgresElasticClient type that implements the fullElasticFacade.ElasticClientinterface using PostgreSQL — eliminating the need for a separate ElasticSearch/OpenSearch cluster.Motivation
Many Moqui deployments already run PostgreSQL. Requiring a separate ElasticSearch/OpenSearch cluster adds operational complexity, memory overhead, and cost — especially for small-to-medium deployments. This provides a zero-dependency alternative that uses PostgreSQL's native JSONB, tsvector full-text search, and GIN indexes.
Changes
New Files
PostgresElasticClient.groovy— FullElasticClientimplementation: index/get/update/delete/bulk/search/count operations backed by PostgreSQL tablesElasticQueryTranslator.groovy— Translates ElasticSearch Query DSL (bool,term,terms,range,nested,exists,match_all,query_string,ids) into parameterized PostgreSQL SQLPostgresSearchLogger.groovy— Log4j2 appender that writes structured logs to PostgreSQL instead of ESSearchEntities.xml— Moqui entity definitions formoqui_search_index,moqui_document,moqui_logs,moqui_http_logwith JSONB columns, tsvector, and GIN/BRIN indexesmoqui-postgres-only-compose.yml— Docker Compose for postgres-only deploymentModified Files
ElasticFacadeImpl.groovy— Addedtype="postgres"client instantiation pathElasticRequestLogFilter.groovy— Updated to work with postgres clientMoquiDefaultConf.xml— Default configuration for postgres searchmoqui-conf-3.xsd— Schema update fortypeattribute onelastic-clientbuild.gradle— PostgreSQL JDBC driver dependencyMoquiSuite.groovy— Test suite registrationTests (83 tests, all passing)
PostgresSearchTranslatorTests.groovy— 46 unit tests for query translation including 13 SQL injection prevention testsPostgresElasticClientTests.groovy— 37 integration tests for CRUD, bulk, search, count, delete-by-queryPostgresSearchSuite.groovy— JUnit test suiteSecurity
^[a-zA-Z0-9_@][a-zA-Z0-9_.\-]*$regex before SQL interpolation--) SQL comments explicitly blockedConfiguration
Set
type="postgres"on anyelastic-clientelement in your Moqui XML configuration. The client will use thetransactionalentity group's database connection.Testing
All 166 Postgres-related tests pass (83 per suite × 2 suite runs). No regressions in existing framework tests.