feat: Cognitive Search infrastructure with env-based switching, mock LiQE/Lunr engine, Storybook fixes, and CI hardening#237
Draft
arif-u-ahmed wants to merge 163 commits intomainfrom
Draft
Conversation
- Created @cellix/mock-cognitive-search package with in-memory implementation - Created @sthrift/service-cognitive-search package with auto-detection - Added domain interfaces and search index definitions - Implemented GraphQL schema and resolvers for item listing search - Added service registration and environment configuration - Core search functionality working end-to-end - Event handlers temporarily disabled due to build issues TODO: Fix module resolution issues for event handlers TODO: Complete integration tests and documentation
…zure support - Add @cellix/mock-cognitive-search package with in-memory Azure-like API - Add @sthrift/service-cognitive-search with environment-driven implementation selection - Implement domain events for item listing updates/deletions - Add search index helpers with retry logic and hash-based change detection - Integrate ServiceCognitiveSearch into API service registry - Configure environment variables for mock/Azure switching Foundation for cognitive search with mock implementation complete. Real Azure integration and comprehensive testing pending.
…lback - Add AzureCognitiveSearch client with full Azure SDK integration - Implement automatic switching between Azure and mock implementations - Add comprehensive documentation and Azure vs mock comparison example - Fix event handler integration with proper transaction handling - Add Azure environment variable configuration support - Ensure graceful fallback to mock when Azure credentials unavailable
- Add comprehensive data seeding service to mock-mongodb-memory-server - Seed 5 mock users and 6 mock item listings with proper ObjectId relationships - Create database-driven cognitive search example replacing hardcoded samples - Integrate ItemListingSearchApplicationService into application services - Add cognitive search to existing GraphQL resolvers with fallback support - Ensure all mock data follows ShareThrift domain model and is interconnected - Add MongoDB indexes for performance and proper data relationships - Update documentation to reflect database-driven approach All mock data is now seeded into MongoDB and queried from database, satisfying requirements for interconnected, non-random data structure.
- Add item-listing-search.graphql schema definition - Add azure-vs-mock-comparison.ts example for cognitive search - Clean up remaining untracked files from development
- Add dist/ folders to .gitignore - Add *.tsbuildinfo to .gitignore - Add test-*.js files to .gitignore - Add *.json test results to .gitignore
- Add LunrSearchEngine wrapper with TF-IDF relevance scoring - Implement field boosting (title 10x, description 2x) - Add wildcard and fuzzy matching support - Include comprehensive test suite (20 tests) - Add JSDoc documentation following CellixJS standards - Maintain Azure Cognitive Search API compatibility - Fix query processing and consistent count handling
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The pull request #237 has too many files changed.
The GitHub API will only let us fetch up to 300 changed files, and this pull request has 1160.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Created @cellix/mock-cognitive-search package with in-memory implementation - Created @sthrift/service-cognitive-search package with auto-detection - Added domain interfaces and search index definitions - Implemented GraphQL schema and resolvers for item listing search - Added service registration and environment configuration - Core search functionality working end-to-end - Event handlers temporarily disabled due to build issues TODO: Fix module resolution issues for event handlers TODO: Complete integration tests and documentation
…zure support - Add @cellix/mock-cognitive-search package with in-memory Azure-like API - Add @sthrift/service-cognitive-search with environment-driven implementation selection - Implement domain events for item listing updates/deletions - Add search index helpers with retry logic and hash-based change detection - Integrate ServiceCognitiveSearch into API service registry - Configure environment variables for mock/Azure switching Foundation for cognitive search with mock implementation complete. Real Azure integration and comprehensive testing pending.
…lback - Add AzureCognitiveSearch client with full Azure SDK integration - Implement automatic switching between Azure and mock implementations - Add comprehensive documentation and Azure vs mock comparison example - Fix event handler integration with proper transaction handling - Add Azure environment variable configuration support - Ensure graceful fallback to mock when Azure credentials unavailable
- Add comprehensive data seeding service to mock-mongodb-memory-server - Seed 5 mock users and 6 mock item listings with proper ObjectId relationships - Create database-driven cognitive search example replacing hardcoded samples - Integrate ItemListingSearchApplicationService into application services - Add cognitive search to existing GraphQL resolvers with fallback support - Ensure all mock data follows ShareThrift domain model and is interconnected - Add MongoDB indexes for performance and proper data relationships - Update documentation to reflect database-driven approach All mock data is now seeded into MongoDB and queried from database, satisfying requirements for interconnected, non-random data structure.
- Add item-listing-search.graphql schema definition - Add azure-vs-mock-comparison.ts example for cognitive search - Clean up remaining untracked files from development
- Add dist/ folders to .gitignore - Add *.tsbuildinfo to .gitignore - Add test-*.js files to .gitignore - Add *.json test results to .gitignore
- Add LunrSearchEngine wrapper with TF-IDF relevance scoring - Implement field boosting (title 10x, description 2x) - Add wildcard and fuzzy matching support - Include comprehensive test suite (20 tests) - Add JSDoc documentation following CellixJS standards - Maintain Azure Cognitive Search API compatibility - Fix query processing and consistent count handling
- Add 27 tests for AzureCognitiveSearch implementation - Constructor and authentication (API key + DefaultAzureCredential) - Service lifecycle (startup/shutdown) - Index management (create, delete, exists) - Document operations (index, delete with proper error handling) - Search operations with facets and filtering - Field type conversion (all Edm types) - Field attribute handling (searchable, filterable, etc.) - Add 12 tests for ServiceCognitiveSearch wrapper - Environment detection (USE_MOCK_SEARCH, USE_AZURE_SEARCH) - Service lifecycle - Proxy method delegation - Graceful fallback handling - Add comprehensive test documentation - TESTING_README.md: Quick start guide and commands - TEST_DOCUMENTATION.md: Detailed test descriptions (850+ lines) - TEST_EXECUTION_SUMMARY.md: Execution summary and coverage - Configure vitest for test execution - vitest.config.ts with coverage settings - Proper vi.hoisted() pattern for mock declarations - Type-safe imports from @cellix/mock-cognitive-search Test Results: 39/39 passing (100% success rate) Mock Coverage: Full Azure SDK isolation (@azure/search-documents, @azure/identity) Type Safety: All TypeScript errors resolved with proper type assertions Note: Identified bug in deleteDocument implementation (documented in test)
arif-u-ahmed
force-pushed
the
feature/mock-cognitive-search-service
branch
from
6cf862f to
dd1dd04
Compare
- Resolved conflicts in package.json files (migrated to pnpm) - Resolved conflicts in apps/api/src/index.ts (kept cognitive search service) - Resolved conflicts in .gitignore (combined build artifacts) - Removed deleted files (package-lock.json, tsconfig.tsbuildinfo) - Updated workspace dependencies to use workspace:* syntax - Preserved cognitive search functionality while adopting main's changes
arif-u-ahmed
had a problem deploying
to
copilot
GitHub Actions
Failure
…add safety-cap and linear parser; add ambient types for `liqe` and narrow parsed.type access
- Access raw props directly instead of domain entity getters to avoid crashes - Add graceful fallback for missing nested data (sharer.account.profile) - Extract value objects properly (e.g., description.value) - Populate sharer field in bulkIndexListings query - Add detailed logging for troubleshooting indexing issues Fixes issue where bulkIndexListings failed with 'Cannot read properties of undefined' errors when domain entities had unpopulated nested fields. Result: Successfully indexes 10/10 listings with proper error handling.
- Implement automatic search indexing via integration events - ItemListing.onSave() raises ItemListingUpdatedEvent and ItemListingDeletedEvent - Event handlers automatically update search index on create/update/delete - No manual reindexing needed during runtime - Fix architectural duplication (clean architecture) - Remove duplicate ListingSearchIndexSpec from search-service-index - Make infrastructure layer import from domain layer - Establish domain as single source of truth for business logic - Fix converter to handle incomplete domain entities - Access raw props to avoid undefined nested properties - Handle both domain entities and test objects - Extract values from value objects properly - Clean up diagnostic logging - Remove console.log statements from production code - Keep only error logging for troubleshooting - Fix build configuration - Add runtime export to types-only packages - Switch to node.json base config for correct output paths - Fix package.json exports to point to dist/src/ - Add TypeScript project references - Add comprehensive documentation - documents/automatic-search-indexing.md: Architecture and usage guide - documents/test-automatic-indexing.md: Step-by-step testing scenarios All 2924 tests passing. All 29 packages build successfully.
The test was still importing from the deleted indexes/ directory. Now imports ListingSearchIndexSpec from @sthrift/domain instead.
…vice - 30 test cases covering all methods and edge cases - Tests for searchListings with various filters (category, state, location, date range, sharerId) - Tests for pagination, sorting, and facets - Tests for bulkIndexListings including error handling - Tests for edge cases (null values, empty arrays, etc.) - Achieves 100% code coverage for listing-search.ts
- listing-search.feature: 33 scenarios covering search, filtering, pagination, bulk indexing - service-search-index.feature: 31 scenarios covering facade operations, CRUD, search - item-listing.feature: Added 5 scenarios for onSave() integration event behavior Feature files document behavior in Gherkin format for better understanding and alignment with existing codebase documentation patterns.
… scenarios - Implemented 5 vitest-cucumber scenario tests for ItemListing onSave() behavior - Scenario: Raising integration event when listing is modified - Scenario: Not raising update event when listing is not modified - Scenario: Raising integration event when listing is deleted - Scenario: Prioritizing delete event over update event - Uses instanceof checks for ItemListingUpdatedEvent and ItemListingDeletedEvent - All 257 tests passing (22ms) Fixes Azure Pipeline ScenarioNotCalledError for domain tests
…tory The ListingsPageContainer now uses both itemListings and searchListings GraphQL queries depending on whether filters/search are active. The Storybook stories were only mocking the itemListings query, causing Apollo MockLink to fail with module loading errors in CI. Root cause: - Container uses conditional query logic (shouldUseSearch flag) - When no search/filter: queries itemListings - When searching/filtering: queries searchListings - Story only mocked itemListings, causing unmocked query error Changes: - Import ListingsPageSearchListingsDocument from generated types - Add searchListings query mock to meta.parameters.apolloClient.mocks - Include proper GraphQL variables structure (searchString, options, filter) - Add __typename fields for Apollo cache normalization - Include sharerName/sharerId fields required by search document - Add facets mock data (category, state, sharerId) - Update EmptyListings story with empty search results mock - Remove unnecessary async from Loading story play function Follows codebase patterns: - Multiple query mocking (same as reservations-view-active.stories) - Proper __typename usage for cache normalization - Co-located with container and .graphql files - Uses withMockApolloClient decorator pattern Fixes CI test failures: - Test Files: 1 failed | 100 passed (101) - Tests: 5 failed | 552 passed (557) - Error: 'Failed to fetch dynamically imported module' All backend tests continue passing: - Domain: 3083 tests ✓ - Application Services: 932 tests ✓
Contributor
There was a problem hiding this comment.
Sorry @jasonmorais, your pull request is larger than the review limit of 150000 diff characters
…e/mock-cognitive-search-service - Resolved conflicts keeping cognitive search service alongside messaging service refactor - Updated package manager from npm to pnpm (workspace:* dependencies) - Added AppealRequest context to DomainDataSource and passport classes - Renamed twilioConversationId to messagingConversationId in conversation test - Removed deleted build artifacts (tsconfig.tsbuildinfo, package-lock.json)
Resolves remaining conflict in conversation.ts getNewInstance method, taking the refactored setter-based assignment pattern from main.
Resolved conflicts by: - Using main's refactored package names (@cellix/* and @apps/api) throughout - Retaining @sthrift/service-cognitive-search dependency in apps/api and event-handler - Merging cognitive search into types/item-listing.resolvers.ts (myListingsAll fallback) - Keeping searchService in ApiContextSpec (context-spec) - Keeping cognitive-search and events exports in domain/index.ts - Accepting deletions of conversation.test.ts and listing/item-listing.resolvers.ts - Removing outdated listing/item-listing-search.* files (superseded by types/ versions) - Adding missing UserEntityReference import in conversation.entity.ts - Adding _accountPlanPassport field to guest/system passport classes
Resolved conflicts from remote feature branch which refactored: - Renamed @cellix/mock-cognitive-search -> @sthrift/search-service-index - Adopted @cellix/search-service + @sthrift/search-service-index architecture - Payment service renamed to @cellix/payment-service with @sthrift/payment-service-* impls - Messaging service uses @cellix/messaging-service - Event handlers use RegisterIntegrationEventHandlers pattern - Application services use ListingSearchApplicationService (renamed from ItemListingSearch) - Domain interfaces updated: ListingSearchDocument/Input/Result (renamed from ItemListing*) - Restored listing/item-listing.resolvers.ts from remote (cleaner implementation) - Removed node_modules and dist artifacts from search-service-index tracking
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Replace ambiguous optional-quote patterns ['"']?([^'"]+)['"']? with unambiguous alternation branches to eliminate O(n^2) backtracking. Fixes CodeQL js/polynomial-redos high severity alerts on lines 85, 169, 173.
Replace exec() loop over user-controlled filterString with plain string operations (indexOf/slice/trim) so no regex is ever applied to tainted data. /\s+and\s+/i (for splitting) and /^\w+$/ (for field validation) are both safe: their quantifiers do not create overlapping NFA paths. Resolves CodeQL js/polynomial-redos High alerts on liqe-filter-engine.ts.
/\s+and\s+/i on user input causes O(n^2) work on strings of all spaces because the engine retries \s+ at every position with every length. Replace it with toLowerCase+indexOf to split on ' and ' without any regex applied to the tainted filterString.
Multiple package.json files referenced non-existent workspace package
names, causing ERR_PNPM_LOCKFILE_CONFIG_MISMATCH and preventing the
pipeline's pnpm install step from succeeding. Downstream Playwright
steps failed only because install never completed.
Changes:
- apps/api/package.json: replace @cellix/payment-service → @cellix/service-payment-base,
@sthrift/payment-service-{mock,cybersource} → @cellix/service-payment-{mock,cybersource},
@sthrift/messaging-service-{mock,twilio} → @cellix/service-messaging-{mock,twilio},
@sthrift/service-blob-storage → @cellix/service-blob,
@sthrift/service-{mongoose,otel,token-validation} → @cellix/service-{mongoose,otel,token-validation}
- apps/api/src/index.ts: update payment import paths and class names
(PaymentServiceMock/Cybersource → ServicePaymentMock/Cybersource)
- context-spec/package.json + src: fix @cellix/payment-service →
@cellix/service-payment-base, @cellix/messaging-service →
@cellix/service-messaging-base, @sthrift/service-token-validation →
@cellix/service-token-validation
- service-cognitive-search: replace defunct @cellix/mock-cognitive-search
with @cellix/search-service + @sthrift/search-service-index; update
CognitiveSearchBase/Service → SearchService, startup/shutdown → startUp/shutDown
- graphql item-listing-search.resolvers.ts: remove dead import of
@sthrift/service-cognitive-search; use applicationServices.Listing.ListingSearch
- Remove accidentally-tracked node_modules from service-cognitive-search
- Regenerate pnpm-lock.yaml
@graphql-tools/code-file-loader (CJS build) calls require() on the
schema TypeScript file. In an ESM package ("type":"module"), .ts files
are treated as ES modules which cannot be require()'d, causing:
Error: Exported schema must be of type GraphQLSchema, text, AST, or
introspection JSON.
The file's own comment says "ensure these remain as require statements
as they get called from graphql-code-generator". Renaming to .cts
(CommonJS TypeScript) makes TypeScript compile it as CommonJS regardless
of the package module type, so require() succeeds.
Also updates codegen.yml and knip.json to reference the .cts extension.
Fixes: turbo //:gen failure → build failure → test:arch failure → knip
failure (knip fails only because vitest-config is never built when gen
fails upstream).
Resolve the remaining CI failures by fixing GraphQL schema loading, removing stale/legacy resolver collisions, restoring broken merged test content, and aligning package imports/dependencies with current renamed service packages. Also: - remove stale ui-components dist .tsx artifacts that caused downstream TS resolution errors in the UI app build - make ui app build ignore story files in tsc - add missing search-service-index runtime deps (liqe, lunr) - update knip ignore list for known fixture/legacy files Verified locally: - pnpm run gen - pnpm run test:arch - pnpm knip
Update search-index handlers to use ListingSearchIndexSpec and convertListingToSearchDocument so event-handler builds against the current domain API in CI.
Make shared arch-unit timeout thresholds configurable and increase timeouts for the heavier sthrift dependency and member-ordering checks so turbo test:arch completes reliably in CI.
Remove leftover conflict markers from index barrel exports, keep facade and legacy exports aligned with consumers, and add @types/lunr so tsc passes in CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Overview
This PR implements the end-to-end Cognitive Search infrastructure and associated developer experience improvements. It adds an environment-aware search service that can switch between Azure Cognitive Search and a local mock implementation, introduces a LiQE-based filter engine, strengthens CI reliability (pnpm lockfile, Playwright browsers), and fixes Storybook states for Reservations.
What’s included
ServiceCognitiveSearchthat selects Azure or Mock at runtime via env vars@sthrift/service-cognitive-searchintegration with@cellix/api-services-specliqeambient types and safe narrowingComponentQueryLoader.errorComponent@cellix/*deps toworkspace:*, refreshedpnpm-lock.yamlpnpm dlx playwright@latest install --with-depsConfiguration
SEARCH_SERVICE_MODE=azureAZURE_SEARCH_ENDPOINT=...AZURE_SEARCH_API_KEY=...SEARCH_SERVICE_MODE=mockVerification
Security/Performance
Breaking changes
Notes
Summary by Sourcery
Implement end-to-end Cognitive Search infrastructure with environment-based switching between Azure and in-memory mock, integrate search into application services and GraphQL API, and improve developer experience and CI reliability.
New Features:
Enhancements:
Build:
Tests: