Skip to content

awakari/interests

BranchesTags

Repository files navigation

Contents

  1. Overview
    1.1. Purpose
    1.2. Definitions
       1.2.1. Condition
          1.2.1.1. Group Condition
          1.2.1.2. Key Condition
          1.2.1.3. Text Condition
       1.2.2. Interest
  2. Configuration
  3. Deployment
    3.1. Prerequisites
    3.2. Bare
    3.3. Docker
    3.4. K8s
       3.4.1. Helm
  4. Usage
    4.1. Create
    4.2. Read
    4.3. Update
    4.4. Delete
    4.5. Search
       4.5.1. By Condition
       4.5.2. By Account
  5. Design
    5.1. Requirements
    5.2. Approach
       5.2.1. Data Schema
          5.2.1.1. Interest
          5.2.1.2. Group Condition
          5.2.1.3. Text Condition
    5.2. Limitations
  6. Contributing
    6.1. Versioning
    6.2. Issue Reporting
    6.3. Building
    6.4. Testing
       6.4.1. Functional
       6.4.2. Performance
    6.5. Releasing

1. Overview

Interests storage service.

1.1. Purpose

The purpose is to serve both public interests API (CRUD + Search) and internal Search.

The main internal function of the service is to find all interests by a condition. For example, a message metadata is matching a certain pattern. Then it's necessary to find all interests those have this pattern in the corresponding condition.

1.2. Definitions

1.2.1. Condition

A condition represents a message matching criteria. The common property for any type of condition is a negation flag (Not). When the negation flag set to true the condition is treated as a negation, otherwise it's a proposition.

1.2.1.1. Group Condition

A group condition represents a group of child conditions coupled with a certain logic: And, Or, Xor.

1.2.1.2. Key Condition

A key condition is an abstract condition specifying a key that should match the message metadata key. Also has a unique condition id.

1.2.1.3. Text Condition

A text condition is a key condition containing the terms for a text search.

1.2.1.4. Number Condition

A key condition containing also a number comparison condition.

1.2.2. Interest

Interest is an entity linking the message matching condition with the user account. An interest also has:

  • unique id generated on creation
  • human-readable description

2. Configuration

The service is configurable using the environment variables:

Variable Example value Description
API_PORT 50051 gRPC API port
DB_URI mongodb+srv://localhost/?retryWrites=true&w=majority DB connection URI
DB_NAME interests DB name to store the data
DB_USERNAME interests DB connection username
DB_PASSWORD interests DB connection password
DB_TABLE_NAME interests DB table name to store the data
DB_TABLE_SHARD true Defines whether the service should shard the table on start

3. Deployment

3.1. Prerequisites

A general note is that there should be a MongoDB cluster deployed to be used for storing the pattern data. It's possible to obtain a free cluster for testing purposes using Atlas.

3.2. Bare

Preconditions:

  1. Build patterns executive using make build
  2. Run the conditions-text dependency service

Then run the command:

API_PORT=50051 \
DB_URI=mongodb+srv://localhost/\?retryWrites=true\&w=majority \
DB_NAME=interests \
DB_TABLE_NAME=interests \
./interests

3.3. Docker

alternatively, it's possible to build and run the new docker image in place using the command: (note that the command below requires all env vars to be set in the file env.txt)

make run

3.4. K8s

Prepare the image pull secret:

kubectl create secret docker-registry github-registry \
  --docker-server ghcr.io \
  --docker-username=<GITHUB_USER_NAME> \ 
  --docker-password=<GITHUB_CR_PAT> \
  --docker-email=<USER_EMAIL>

When using external DB, create the corresponding secret:

kubectl create secret generic db-mongo \
  --from-literal=url=<MONGO_URL> \
  --from-literal=username=<MONGO_USERNAME> \
  --from-literal=password=<MONGO_PASSWORD>

3.4.1. Helm

Create a helm package from the sources:

helm package helm/interests/

Install the helm chart:

helm install interests ./interests-<CHART_VERSION>.tgz \
  --values helm/interests/values-db-uri.yaml

where

  • values-db-uri.yaml contains the value override for the DB URI
  • <CHART_VERSION> is the helm chart version

4. Usage

The service provides basic gRPC interface to perform the operation on interests.

4.1. Create

Example command:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -H 'X-Awakari-Group-Id: group0' \
  -H 'X-Awakari-User-Id: user0' \
  -d @ \
  localhost:50051 \
  awakari.interests.Service/Create

Payload:

{
   "description": "my interest 1",
   "enabled": true,
   "cond": {
      "not": false,
      "gc": {
         "logic": 0, 
         "group": [
            {
               "not": false,
               "tc": {
                  "id": "cond0",
                  "key": "key0", 
                  "term": "term0 term1"
               }
            }, 
            {
               "not": true,
               "tc": {
                  "id": "cond1",
                  "key": "key1", 
                  "term": "term2",
                  "exact": true
               }
            }
         ]
      }
   }
}

4.2. Read

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -H 'X-Awakari-Group-Id: group0' \
  -H 'X-Awakari-User-Id: user0' \
  -d '{"id": "17861cda-edc0-4655-be5a-e69a8129aff5"}' \
  localhost:50051 \
  awakari.interests.Service/Read

4.3. Update

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -H 'X-Awakari-Group-Id: group0' \
  -H 'X-Awakari-User-Id: user0' \
  -d @ \
  localhost:50051 \
  awakari.interests.Service/UpdateMetadata

Payload:

{
   "id": "d3911098-99e7-4a69-94f9-3cea0b236a04",
   "description": "my interest 1 updated",
   "enabled": false,
   "cond": {
      "not": false,
      "gc": {
         "logic": 0,
         "group": [
            {
               "not": false,
               "tc": {
                  "id": "cond0",
                  "key": "key0",
                  "term": "term0 term1"
               }
            },
            {
               "not": true,
               "tc": {
                  "id": "cond1",
                  "key": "key1",
                  "term": "term2",
                  "exact": true
               }
            }
         ]
      }
   }
}

4.4. Delete

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -H 'X-Awakari-Group-Id: group0' \
  -H 'X-Awakari-User-Id: user0' \
  -d '{"id": "f7102c87-3ce4-4bb0-8527-b4644f685b13"}' \
  localhost:50051 \
  awakari.interests.Service/Delete

4.5. Search

4.5.1. By Account

The search by account purpose is to be used by a user to find own interests.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/service.proto \
  -H 'X-Awakari-Group-Id: group0' \
  -H 'X-Awakari-User-Id: user0' \
  -d '{"limit": 100, "cursor": "0123456789abcdef"}' \
  localhost:50051 \
  awakari.interests.Service/SearchOwn

4.5.2. By Condition

The search by condition purpose is to be used by a resolver to find the matching interests.

Example:

grpcurl \
  -plaintext \
  -proto api/grpc/private/service.proto \
  -d '{"condId": "14cadd71-c662-4f1a-8b0f-3b17dfb107f5", "limit": 16}' \
  localhost:50051 \
  awakari.interests.private.Service/SearchByCondition

5. Design

5.1. Requirements

# Summary Description
REQ-1 Basic matching Resolve interests matching the input value
REQ-2 Logic Support interest logics for the multiple key-value matches (and, or, not)
REQ-3 Partial matching Support partial (value might be split to lexemes) value matching
REQ-4 Pagination Support query results pagination
REQ-5 On/off switch Support interest disabling for internal query by condition without removing it
REQ-6 Metadata update Support changing a interest description, priority and enabled flag
REQ-7 Authentication Authenticate public API calls: Create, Read, UpdateMetadata, Delete, SearchOwn

5.2. Approach

5.2.1. Data Schema

Interests are stored in the single table under the denormalized schema.

5.2.1.1. Interest

Attribute Type Description
id String Interest ID (generated on creation if not specified)
groupId String User Group Id
userId String User Id
descr String Human readable description
enabled Boolean Defines whether the interest is searchable for a condition matching
expires Time Defines the deadline when the interest may be treated as enabled
created Time
updated Time
cond Condition (currently may be Group or Text) Message matching root immutable criteria

5.2.1.2. Group Condition

Attribute Type Description
not Boolean Defines whether the conditions should act as a negation or not
logic Enum of And/Or/Xor Defines the grouping logic for the child conditions
group Array of child conditions Set of conditions in the group

5.2.1.3. Text Condition

Attribute Type Description
id String Condition UUID (generated on creation)
not Boolean Defines whether the conditions should act as a negation or not
key String Metadata key
term String Text value matching term(s)
exact Boolean Defines whether the condition should match the complete input exactly or not

5.2. Limitations

# Summary Description
LIM-1 TODO TODO

6. Contributing

6.1. Versioning

The service uses the semantic versioning. The single source of the version info is the git tag:

git describe --tags --abbrev=0

6.2. Issue Reporting

TODO

6.3. Building

make build

Generates the sources from proto files, compiles and creates the interests executable.

6.4. Testing

6.4.1. Functional

make test

6.4.2. Performance

TODO

6.5. Releasing

To release a new version (e.g. 1.2.3) it's enough to put a git tag:

git tag -v1.2.3
git push --tags

The corresponding CI job is started to build a docker image and push it with the specified tag (+latest).

About

Subscriptions storage

Resources

Readme

License

AGPL-3.0 license

Contributing

Contributing
Activity
Custom properties

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

1 tags

Packages

 
 
 

Contributors

Languages