[New Hack] 074-AIWithConfluentOnAzure

.devcontainer/074-AIWithConfluentOnAzure/devcontainer.json

@@ -0,0 +1,33 @@
+{
+    "image": "mcr.microsoft.com/devcontainers/python:3.12-bookworm",
+    "workspaceFolder": "/workspace/074-AIWithConfluentOnAzure/Student/Resources",
+	"workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind,consistency=cached",
+    "hostRequirements": {
+      "cpus": 2,
+      "memory": "4gb"  
+    },
+    "waitFor": "onCreateCommand",
+    "updateContentCommand": "python3 -m pip install -r requirements.txt",
+    "postCreateCommand": "",
+    "features": {
+        "ghcr.io/devcontainers/features/docker-in-docker:2": {},
+        "ghcr.io/devcontainers/features/azure-cli:1": {},
+        "ghcr.io/devcontainers/features/terraform:1": {}
+      },
+    "customizations": {
+      "codespaces": {
+        "openFiles": []
+      },
+      "vscode": {
+        "extensions": [
+          "ms-toolsai.jupyter",
+          "ms-python.python",
+          "github.copilot",
+          "github.copilot-chat",
+          "HashiCorp.terraform",
+          "humao.rest-client",
+          "ms-azuretools.vscode-docker"  
+        ]
+      }
+    }
+  }

.gitignore

@@ -333,4 +333,35 @@ ASALocalRun/
 .mfractor/
 .DS_Store
 
-.env
+.env
+
+
+shared_environment_variables.sh
+shared_environment_variables.env
+
+.env.sh
+
+# Confluent Ignores
+*confluent-environment*
+*confluent-environment.sh
+
+# Terraform Stuff
+terraform.tfstate.backup
+.terraform
+*.tfstate
+.terraform*
+azure_environment_variables.sh
+main_variables.tf
+microsoft_hackathon.plan
+
+# Othe files from Python and Node
+.cosmos_permissions
+DRAFT.txt
+node_modules
+.idea
+.vscode
+.project
+__pycache__
+*.final.json
+
+

074-AIWithConfluentOnAzure/.wordlist.txt

@@ -0,0 +1,8 @@
+Flink
+DDL
+DML
+Confluent
+Confluent's
+ksqlDB
+replenishments
+upsert

074-AIWithConfluentOnAzure/Coach/README.md

@@ -0,0 +1,71 @@
+# What The Hack - AI With Confluent On Azure - Coach Guide
+
+## Introduction
+
+Welcome to the coach's guide for the AI With Confluent On Azure What The Hack. Here you will find links to specific guidance for coaches for each of the challenges.
+
+This hack includes an optional [lecture presentation](Lectures.pptx) that features short presentations to introduce key topics associated with each challenge. It is recommended that the host present each short presentation before attendees kick off that challenge.
+
+**NOTE:** If you are a Hackathon participant, this is the answer guide. Don't cheat yourself by looking at these during the hack! Go learn something. :)
+
+## Coach's Guides
+
+- Challenge 00: **[Prerequisites - Ready, Set, GO!](./Solution-00.md)**
+	 - Prepare Azure and Confluent Cloud environments with required infrastructure
+- Challenge 01: **[Build the Flink Data Pipeline](./Solution-01.md)**
+	 - Create streaming data pipeline that merges multiple data streams using Apache Flink
+- Challenge 02: **[Supplier Experience](./Solution-02.md)**
+	 - Interact with the AI agent as a supplier to query and replenish inventory
+- Challenge 03: **[Customer Experience](./Solution-03.md)**
+	 - Simulate customer purchases and returns with real-time inventory updates
+- Challenge 04: **[Employee Experience](./Solution-04.md)**
+	 - Access comprehensive operational insights across all transactions
+
+## Coach Prerequisites
+
+This hack has pre-reqs that a coach is responsible for understanding and/or setting up BEFORE hosting an event. Please review the [What The Hack Hosting Guide](https://aka.ms/wthhost) for information on how to host a hack event.
+
+The guide covers the common preparation steps a coach needs to do before any What The Hack event, including how to properly configure Microsoft Teams.
+
+### Student Resources
+
+Before the hack, it is the Coach's responsibility to download and package up the contents of the `/Student/Resources` folder of this hack into a "Resources.zip" file. The coach should then provide a copy of the Resources.zip file to all students at the start of the hack.
+
+Always refer students to the [What The Hack website](https://aka.ms/wth) for the student guide: [https://aka.ms/wth](https://aka.ms/wth)
+
+**NOTE:** Students should **not** be given a link to the What The Hack repo before or during a hack. The student guide does **NOT** have any links to the Coach's guide or the What The Hack repo on GitHub.
+
+### Additional Coach Prerequisites
+
+- Ensure you have a Confluent Cloud account with permissions to create Kafka clusters, topics, and connectors.
+- Pre-provision or verify access to the required Azure services: Azure OpenAI, Azure Cosmos DB, Azure AI Search, Azure Redis Cache, and Azure Storage Account.
+- Test the Terraform modules in the `Student/Resources/terraform` folder to ensure they deploy successfully in your target environment.
+
+## Azure Requirements
+
+This hack requires students to have access to an Azure subscription where they can create and consume Azure resources. These Azure requirements should be shared with a stakeholder in the organization that will be providing the Azure subscription(s) that will be used by the students.
+
+- Azure subscription with Owner access
+- Ability to create the following resources: Azure OpenAI, Azure Cosmos DB, Azure AI Search, Azure Redis Cache, Azure Storage Account
+- Sufficient quota for Azure OpenAI model deployments
+
+## Suggested Hack Agenda
+
+- Challenge 00 (1 hour) - Prerequisites and infrastructure provisioning
+- Challenge 01 (1.5 hours) - Build the Flink Data Pipeline
+- Challenge 02 (1 hour) - Supplier Experience
+- Challenge 03 (1 hour) - Customer Experience
+- Challenge 04 (1 hour) - Employee Experience
+
+## Repository Contents
+
+- `./Coach`
+  - Coach's Guide and related files
+- `./Coach/Solutions`
+  - Solution files with completed example answers to a challenge
+- `./Coach/Lectures.pptx`
+  - Optional lecture presentations for each challenge
+- `./Student`
+  - Student's Challenge Guide
+- `./Student/Resources`
+  - Terraform modules, configuration scripts, AI agent application code, and connector configurations meant to be provided to students (must be packaged up by the coach and provided to students at start of event)

074-AIWithConfluentOnAzure/Coach/Solution-00.md

@@ -0,0 +1,18 @@
+# Challenge 00 - Prerequisites - Ready, Set, GO! - Coach's Guide 
+
+**[Home](./README.md)** - [Next Solution >](./Solution-01.md)
+
+## Notes & Guidance
+
+Ensure students have all prerequisites installed before proceeding:
+
+- Verify students have Azure CLI, Terraform CLI, and Confluent CLI installed and authenticated
+- Students should extract the Resources.zip file and navigate to the Terraform directory
+- The Terraform modules require Azure subscription ID, principal IDs, and Confluent API keys to be set in the variables files
+- Run `terraform init` followed by `terraform plan` and `terraform apply` to deploy all infrastructure
+- Common issues:
+  - Insufficient Azure permissions or quota limits for Azure OpenAI
+  - Missing or incorrect Confluent Cloud API keys
+  - Terraform state lock issues if multiple students share a subscription
+- After deployment, verify all connectors are in a "Running" state in Confluent Cloud
+- Confirm the MCP-powered AI agent responds when prompted with its name and can list the eight grocery store departments

074-AIWithConfluentOnAzure/Coach/Solution-01.md

@@ -0,0 +1,21 @@
+# Challenge 01 - Build the Flink Data Pipeline - Coach's Guide 
+
+[< Previous Solution](./Solution-00.md) - **[Home](./README.md)** - [Next Solution >](./Solution-02.md)
+
+## Notes & Guidance
+
+Students need to connect to the Apache Flink SQL editor in Confluent Cloud and create Flink tables mapped to existing Kafka topics.
+
+- Students must create Flink tables for all six topics: `product_sku`, `product_pricing`, `product_departments`, `purchases`, `returns`, and `replenishments`
+- The merge logic for `net_sales` should:
+  - Add purchase amounts to the running total per SKU
+  - Subtract return amounts from the running total per SKU
+- The merge logic for `net_inventory_count` should:
+  - Subtract purchase quantities from inventory per SKU
+  - Add replenishment quantities to inventory per SKU
+  - For returns: only add back to inventory if the product belongs to the appliance department; ignore returns from all other departments
+- Reference Flink SQL solutions are available in the `Coach/Solutions/flink-sql` folder
+- Common issues:
+  - Students may forget to handle the appliance department return logic
+  - Schema mismatches between Kafka topic schemas and Flink table definitions
+  - Flink SQL statements must be running continuously; verify they are active in the Confluent Cloud UI

074-AIWithConfluentOnAzure/Coach/Solution-02.md

@@ -0,0 +1,14 @@
+# Challenge 02 - Supplier Experience - Coach's Guide 
+
+[< Previous Solution](./Solution-01.md) - **[Home](./README.md)** - [Next Solution >](./Solution-03.md)
+
+## Notes & Guidance
+
+Students interact with the supplier agent to query and replenish inventory.
+
+- The supplier agent should display real-time inventory levels per SKU and per department
+- When a student replenishes inventory, the change should be reflected in the `net_inventory_count` table and subsequently in Azure AI Search
+- Verify students confirm that the AI agent retrieves updated values from MCP services after replenishment actions
+- Common issues:
+  - Delays in data propagation from Kafka through Flink to Azure AI Search; allow a few seconds for updates
+  - If the agent returns stale data, check that the Flink SQL merge statements are running and the sink connector to Azure AI Search is active

074-AIWithConfluentOnAzure/Coach/Solution-03.md

@@ -0,0 +1,15 @@
+# Challenge 03 - Customer Experience - Coach's Guide 
+
+[< Previous Solution](./Solution-02.md) - **[Home](./README.md)** - [Next Solution >](./Solution-04.md)
+
+## Notes & Guidance
+
+Students simulate customer purchases and returns and observe real-time inventory and sales updates.
+
+- Students should record inventory levels before and after purchases/returns to validate the pipeline
+- After a purchase, inventory should decrease and net sales should increase
+- After a return, net sales should decrease; inventory should only increase if the returned product belongs to the appliance department
+- The three-second update window is a guideline; slight delays are acceptable but data should converge quickly
+- Common issues:
+  - Students may not observe updates if the sink connector to Azure AI Search has fallen behind
+  - Remind students to check both the `net_sales` and `net_inventory_count` values after each action

074-AIWithConfluentOnAzure/Coach/Solution-04.md

@@ -0,0 +1,16 @@
+# Challenge 04 - Employee Experience - Coach's Guide 
+
+[< Previous Solution](./Solution-03.md) - **[Home](./README.md)**
+
+## Notes & Guidance
+
+Students use the employee agent to access comprehensive operational insights across all transactions.
+
+- This challenge requires students to use multiple personas simultaneously (customer, supplier, and employee)
+- The employee agent should display inventory levels, cumulative purchase/return/replenishment units, and net sales for any SKU
+- Students should trigger events from customer and supplier agents and then verify the employee agent reflects those changes
+- All data shown by the employee agent should match what is observed in Azure AI Search and Flink output
+- Common issues:
+  - Students may have difficulty managing multiple browser tabs or windows for different personas
+  - If cumulative values are incorrect, verify the Flink SQL merge logic from Challenge 01 is running correctly
+  - Ensure the sink connector is actively pushing updates to Azure AI Search

074-AIWithConfluentOnAzure/Coach/Solutions/flink-sql/README.md

@@ -0,0 +1,33 @@
+
+## Data Pipeline Overview
+
+The goal of the pipeline is to ensure that 
+- the data from the source data stores (Cosmos DB) flows into Kafka topics
+- the data from the topics are joined and merged into transient tables
+- the data from the transient tables to finalized into the summary topics and tables
+- the data from the summary tables and pushed out to Azure AI search for the AI agent to use via MCP
+
+### Steps to Follow
+
+You will need to run each DDL statement first to set up the Flink Tables.
+
+Note that each Flink table also maps to a Kafka Topic in Confluent cloud. You can browse these topics as well to see the raw data in the topics.
+
+You will then need to run the DML statements to pull data from various topics/tables and push them to the final tables that will go out to Azure AI Search via the sink connectors.
+
+### Location of SQL Statements
+
+The DDL statements are in this folder. Run them one at a time in the Flink SQL Workspace in Confluent Cloud.
+
+It would be nice to run these from the Confluent CLI.
+
+The DML statements that pull from single or multiple sources/topics into the final topics/tables are also available in this folder
+
+### AI Search Sink Configuration
+
+Make sure that the topics in the AI Search Sink connector matches the corresponding final topic/table.
+
+Make sure that the AI Search index name is also correct. The MCP service depends on this correctness to fetch the correct data.
+
+
+

074-AIWithConfluentOnAzure/Coach/Solutions/flink-sql/data_pipeline-ddl.sql

@@ -0,0 +1,50 @@
+
+
+CREATE TABLE `product_inventory_depot` (
+  `key` STRING NOT NULL,
+  `department` STRING NOT NULL,
+  `description` STRING NOT NULL,
+  `inventory_level` DOUBLE NOT NULL,
+  `name` STRING NOT NULL,
+  `sku_id` STRING NOT NULL,
+  `unit_price` DOUBLE NOT NULL,
+   PRIMARY KEY (`key`) NOT ENFORCED 
+)
+DISTRIBUTED BY HASH(`key`) INTO 1 BUCKETS
+WITH (
+  'changelog.mode' = 'upsert',
+  'connector' = 'confluent',
+  'kafka.cleanup-policy' = 'delete',
+  'kafka.compaction.time' = '0 ms',
+  'kafka.max-message-size' = '2097164 bytes',
+  'kafka.retention.size' = '0 bytes',
+  'kafka.retention.time' = '7 d',
+  'key.format' = 'json-registry',
+  'scan.bounded.mode' = 'unbounded',
+  'scan.startup.mode' = 'earliest-offset',
+  'value.format' = 'json-registry'
+);
+
+
+CREATE TABLE `net_sales_depot` (
+  `key` STRING NOT NULL,
+  `department` STRING NOT NULL COMMENT 'Department of the Product',
+  `name` STRING NOT NULL COMMENT 'Name of the Product',
+  `net_sales` DOUBLE NOT NULL COMMENT 'Net sales amount for the transaction.',
+  `sku_id` STRING NOT NULL COMMENT 'Unique identifier for the stock keeping unit (SKU).',
+  PRIMARY KEY (`key`) NOT ENFORCED
+)
+DISTRIBUTED BY HASH(`key`) INTO 1 BUCKETS
+WITH (
+  'changelog.mode' = 'upsert',
+  'connector' = 'confluent',
+  'kafka.cleanup-policy' = 'delete',
+  'kafka.compaction.time' = '0 ms',
+  'kafka.max-message-size' = '2097164 bytes',
+  'kafka.retention.size' = '0 bytes',
+  'kafka.retention.time' = '7 d',
+  'key.format' = 'json-registry',
+  'scan.bounded.mode' = 'unbounded',
+  'scan.startup.mode' = 'earliest-offset',
+  'value.format' = 'json-registry'
+);