From 3d67fc3920ab82e34f9ffa017fcf9cbb926c8663 Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Thu, 7 Aug 2025 11:39:32 -0700 Subject: [PATCH 1/6] Initial commit --- mongo-2-oracle/README.md | 169 +++ mongo-2-oracle/src/orclMongoMigration.sh | 1240 ++++++++++++++++++++++ mongo-2-oracle/step-by-step-guide.md | 270 +++++ 3 files changed, 1679 insertions(+) create mode 100644 mongo-2-oracle/README.md create mode 100644 mongo-2-oracle/src/orclMongoMigration.sh create mode 100644 mongo-2-oracle/step-by-step-guide.md diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md new file mode 100644 index 0000000..03ef019 --- /dev/null +++ b/mongo-2-oracle/README.md @@ -0,0 +1,169 @@ +# MongoDB to Oracle Database 23ai Migration + +**Summary:** Contained herein is a bash script for demonstrating and managing MongoDB to Oracle Database migration in a containerized environment. The `orclMongoMigration.sh` script provides a full-featured environment for demonstrating how to migrate data from MongoDB to Oracle Database. It handles container management, database setup, MongoDB configuration, and the complete migration process in an interactive, user-friendly way. + +## Key Features + +**Container Management:** +* Start/stop Oracle Database containers +* Automatic configuration and environment setup +* Interactive menu system with color-coded output +* Comprehensive command-line interface for automation +* File transfer between host and container +* Volume and resource management + +**Database Setup and Access:** +* Automatic Oracle Database configuration +* Multiple SQL*Plus access modes (nolog, user, SYSDBA) +* ORDS (Oracle REST Data Services) setup and management +* MongoDB initialization and configuration +* MongoDB API compatibility for Oracle + +**MongoDB and Migration:** +* MongoDB instance setup and configuration +* MongoDB shell access via mongosh +* Oracle Database MongoDB API connectivity +* Complete data migration workflow from MongoDB to Oracle +* Sample data generation and import +* Interactive migration process with step-by-step guidance +Demo Application +* Registration demo app deployment +* Sample data generation with geographic features +* Data visualization and testing + +## Prerequisites +* Podman installed (script can install it if not present) +* macOS or Linux operating system +* Internet connection +* 8GB+ RAM recommended for running the Oracle Database container + +## Installation + +1. Download the script + + ```bash + + wget + + ``` + +2. Change permissions to make the script executable + + ```bash + + chmod +x orclMongoMigration.sh + + ``` + +3. Run the script without arguments to display a user-friendly menu with the following sections: + + ```bash + + ./orclMongoMigration.sh + + ``` + + * **Container Management:** - Start/stop Oracle container - Bash and root access - Install utilities and manage volumes - Copy files in and out + * **Database Access & Utilities:** - SQL*Plus connections (nolog, user, SYSDBA) - ORDS setup and management - MongoDB API connectivity + * **MongoDB Operations:** - Start MongoDB instance - mongosh to ORDS or MongoDB + * **Application & Migration:** - Run registration demo app - Add demo data - Migrate data from MongoDB to Oracle + +## Available Commandline Arguments + +* Container Management: + + ```bash + start - Start Oracle container + stop - Stop Oracle container + restart - Restart Oracle container + bash - Bash access to container + root - Root access to container + remove - Remove Oracle container + utils - Install utilities + copyin - Copy file into container + copyout - Copy file out of container + clean - Clean unused volumes + ``` + +* Database Access: + + ```bash + sqlnolog - SQL*Plus nolog connection + sqluser - SQL*Plus user connection + sqlsys - SQL*Plus SYSDBA connection + setupords - Setup ORDS + ords - Start ORDS service + mongoapi - Check MongoDB API connection + ``` + +* MongoDB Commands: + + ```bash + mongodb - Start MongoDB instance + mongoords - Connect to MongoDB via ORDS + mongo - Connect to MongoDB directly + ``` + +* Application Commands: + + ```bash + demoapp - Run Registration Demo App + demodata - Add demo data + migrate - Migrate data + help - Show help message + ``` + +## Migration Workflow Example +Here’s a typical workflow for demonstrating MongoDB to Oracle migration: +1. Start the container: + + ```bash + + ./orclMongoMigration.sh start + + ``` +2. Start MongoDB instance: + ```bash + + ./orclMongoMigration.sh mongodb + + ``` +3. Generate demo data: + ```bash + + ./orclMongoMigration.sh demodata + + ``` +4. Perform the migration: + ```bash + + ./orclMongoMigration.sh migrate + + ``` +5. Verify the migrated data in Oracle: (The script will offer to connect you to SQL*Plus after migration) + +## Container Details +* **Image:** Oracle Database Free (container-registry.oracle.com/database/free) +* **Default Credentials:** +* SYS/SYSTEM password: Oradoc_db1 +* Created user: matt with password matt +* **Exposed Ports:** +* 1521: Oracle Database listener +* 3000: Node.js demonstration app +* 5500: Enterprise Manager Express +* 8080/8443: ORDS (Oracle REST Data Services) +* 27017: MongoDB API compatibility +* 23456: MongoDB native port + +## Troubleshooting +* If the Oracle container fails to start, try running ./orclMongoMigration.sh clean to remove unused volumes +* For migration issues, check that both MongoDB and ORDS are running +* If demo data generation fails, ensure Python libraries are installed correctly +* Check container logs for Oracle Database startup issues + + +## Acknowledgments +* Created by Matt DeMarco (matthew.demarco@oracle.com) +* Oracle Database Free images provided by Oracle + + diff --git a/mongo-2-oracle/src/orclMongoMigration.sh b/mongo-2-oracle/src/orclMongoMigration.sh new file mode 100644 index 0000000..57a8851 --- /dev/null +++ b/mongo-2-oracle/src/orclMongoMigration.sh @@ -0,0 +1,1240 @@ +#!/bin/bash +# +# +#=================================================================================== +# +# FILE: orclMongoMigration.sh +# +# USAGE: ./orclMongoMigration.sh [start|stop|restart|bash|root|sql|ords|mongoapi|help] +# +# DESCRIPTION: Migration demo container moving MongoDB to Oracle +# OPTIONS: See menu or command line arguments +# REQUIREMENTS: Podman, internet connection +# AUTHOR: Matt DeMarco (matthew.demarco@oracle.com) +# CREATED: 04.01.2025 +# VERSION: 1.1 +# +#=================================================================================== + +# Copyright (c) 2025 Oracle and/or its affiliates. + +# The Universal Permissive License (UPL), Version 1.0 + +# Subject to the condition set forth below, permission is hereby granted to any +# person obtaining a copy of this software, associated documentation and/or data +# (collectively the "Software"), free of charge and under any and all copyright +# rights in the Software, and any and all patent rights owned or freely +# licensable by each licensor hereunder covering either (i) the unmodified +# Software as contributed to or provided by such licensor, or (ii) the Larger +# Works (as defined below), to deal in both + +# (a) the Software, and +# (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if +# one is included with the Software (each a "Larger Work" to which the Software +# is contributed by such licensors), + +# without restriction, including without limitation the rights to copy, create +# derivative works of, display, perform, and distribute the Software and make, +# use, sell, offer for sale, import, export, have made, and have sold the +# Software and the Larger Work(s), and to sublicense the foregoing rights on +# either these or other terms. + +# This license is subject to the following condition: +# The above copyright notice and either this complete permission notice or at +# a minimum a reference to the UPL must be included in all copies or +# substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +# Global variables +ORACLE_USER="matt" +ORACLE_PASS="matt" +ORACLE_SYS_PASS="Oradoc_db1" +ORACLE_CONTAINER="MoMoMatt" +ORACLE_PDB="FREEPDB1" +NETWORK_NAME="demonet" #set network +CONTAINER_PORT_MAP="-p 1521:1521 -p 3000:3000 -p 5902:5902 -p 5500:5500 -p 8000:8000 -p 8080:8080 -p 8443:8443 -p 27017:27017 -p 23456:23456" +MAX_INVALID=3 +INVALID_COUNT=0 + +# Colors for better readability +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +CYAN='\033[0;36m' +NC='\033[0m' # No Color + + + +#=========================== +# Helper Functions +#=========================== + +logInfo() { + echo -e "${GREEN}[INFO]${NC} $1" +} + +logWarning() { + echo -e "${YELLOW}[WARNING]${NC} $1" +} + +logError() { + echo -e "${RED}[ERROR]${NC} $1" +} + +logSuccess() { + echo -e "${GREEN}[SUCCESS]${NC} $1" +} + +# Display formatted menu +displayMenu() { + clear + echo -e "${CYAN}===========================================================================${NC}" + echo -e "${CYAN} Oracle Database & MongoDB Migration Demo Toolkit ${NC}" + echo -e "${CYAN}===========================================================================${NC}" + + echo -e "\n${GREEN}Container Management:${NC}" + echo " 1) Start Oracle container 6) Install utilities" + echo " 2) Stop Oracle container 7) Copy file into container" + echo " 3) Bash access 8) Copy file out of container" + echo " 4) Root access 9) Clean unused volumes" + echo " 5) Remove Oracle container 10) Exit script" + + echo -e "\n${GREEN}Database Access & Utilities:${NC}" + echo " 11) SQL*Plus nolog connection 14) Setup ORDS" + echo " 12) SQL*Plus user connection 15) Start ORDS service" + echo " 13) SQL*Plus SYSDBA connection 16) Check MongoDB API connection" + + echo -e "\n${GREEN}MongoDB Operations:${NC}" + echo " 17) Start MongoDB instance 19) mongosh to MongoDB" + echo " 18) mongosh to MongoDB API" + + echo -e "\n${GREEN}Application & Migration:${NC}" + echo " 20) Run Registration Demo App 22) Migrate data" + echo " 21) Add demo data" + + echo -e "\n${CYAN}===========================================================================${NC}" + read -p "Please enter your choice [1-22]: " menuChoice + export menuChoice=$menuChoice +} + +# Check if podman is installed and running +checkPodman() { + if ! command -v podman > /dev/null 2>&1; then + logError "Podman is not installed on your system." + + read -p "Would you like to install Podman and its dependencies? (y/n): " choice + if [[ "$choice" =~ ^[Yy]$ ]]; then + logInfo "Installing Podman and dependencies..." + + # Install Homebrew if it's not installed + if ! command -v brew > /dev/null 2>&1; then + logInfo "Homebrew is not installed. Installing Homebrew first..." + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + export PATH="/usr/local/bin:$PATH" # For Intel Macs + export PATH="/opt/homebrew/bin:$PATH" # For Apple Silicon + fi + + # Install Podman, QEMU, and vfkit + brew tap cfergeau/crc + brew install vfkit qemu podman podman-desktop + + # Initialize Podman machine + logInfo "Initializing Podman machine..." + podman machine init --cpus 8 --memory 16384 --disk-size 550 + + logInfo "Starting Podman machine..." + podman machine start + + logSuccess "Podman installation complete!" + else + logError "Podman is required to run this script. Exiting..." + exit 1 + fi + fi + + # Verify Podman is running + if ! podman ps > /dev/null 2>&1; then + logInfo "Podman is installed but not running. Starting Podman machine..." + podman machine start + fi +} + +# Create podman network if it doesn't exist +createPodnet() { + if ! podman network inspect $NETWORK_NAME &>/dev/null; then + logInfo "Creating podman network '$NETWORK_NAME'..." + podman network create -d bridge $NETWORK_NAME + fi +} + +# Get running container ID/name +getContainerId() { + export orclRunning=$(podman ps --no-trunc --format "table {{.ID}}\t {{.Names}}\t" | grep -i $ORACLE_CONTAINER | awk '{print $2}') + echo $orclRunning +} + +# Countdown timer +countDown() { + message=${1:-"Please wait..."} + seconds=${2:-3} + + logInfo "$message" + for (( i=$seconds; i>=1; i-- )); do + echo -ne "\rStarting in $i seconds..." + sleep 1 + done + echo -e "\rStarting now! " +} + +# Handle invalid menu choice +badChoice() { + # Increment the invalid choice counter + ((INVALID_COUNT++)) + + logWarning "Invalid choice, please try again..." + logWarning "Attempt $INVALID_COUNT of $MAX_INVALID." + + # Check if invalid attempts exceed the max allowed + if [ "$INVALID_COUNT" -ge "$MAX_INVALID" ]; then + logError "Too many invalid attempts. Exiting the script..." + exit 1 + fi + + sleep 2 +} + +#=========================== +# Core Functions +#=========================== + +# Exit function +doNothing() { + logWarning "You want to quit...yes?" + read -p "Enter yes or no: " doWhat + if [[ $doWhat = yes ]]; then + logInfo "Bye! ¯\\_(ツ)_/¯" + exit 0 + else + return + fi +} + +# List container ports +listPorts() { + container_id=$(getContainerId) + if [ -n "$container_id" ]; then + logInfo "Container ports:" + podman port "$container_id" + + logInfo "Container IP address:" + podman inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$container_id" + else + logError "No running container found." + fi +} + +# Start Oracle container +startOracle() { + checkPodman + createPodnet + + # Check if container is already running + export orclRunning=$(getContainerId) + export orclPresent=$(podman container ls -a --no-trunc --format "table {{.ID}}\t {{.Names}}\t" | grep -i $ORACLE_CONTAINER | awk '{print $2}') + + if [ "$orclRunning" == "$ORACLE_CONTAINER" ]; then + logWarning "Oracle podman container is already running." + listPorts + return + elif [ "$orclPresent" == "$ORACLE_CONTAINER" ]; then + logInfo "Oracle podman container found, restarting..." + podman restart $orclPresent + countDown "Waiting for Oracle to start" 3 + serveORDS + else + echo "Please choose the Oracle Database container version:" + echo "1. Lite Version (Good for general database development)" + echo "2. Full Version (Required for the MongoDB API)" + read -p "Enter your choice [1/2]: " choice + + case $choice in + 1) + image="container-registry.oracle.com/database/free:23.5.0.0-lite" + ;; + 2) + image="container-registry.oracle.com/database/free:latest" + ;; + *) + logWarning "Invalid choice. Defaulting to Full version." + image="container-registry.oracle.com/database/free:latest" + ;; + esac + + logInfo "Provisioning new Oracle container with image: $image" + podman run -d --network="podmannet" $CONTAINER_PORT_MAP -it --name $ORACLE_CONTAINER $image + + if [ $? -ne 0 ]; then + logError "Failed to start Oracle container." + return 1 + fi + + logSuccess "Oracle container started successfully." + countDown "Waiting for Oracle to initialize" 3 + installUtils + fi + listPorts +} + +# Stop Oracle container +stopOracle() { + checkPodman + export stopOrcl=$(podman ps --no-trunc | grep -i oracle | awk '{print $1}') + + if [ -z "$stopOrcl" ]; then + logWarning "No Oracle containers are running." + return + fi + + for i in $stopOrcl; do + logInfo "Stopping container: $i" + podman stop $i + if [ $? -eq 0 ]; then + logSuccess "Container stopped successfully." + else + logError "Failed to stop container." + fi + done + + cleanVolumes +} + +# Clean unused volumes +cleanVolumes() { + logInfo "Cleaning unused volumes..." + podman volume prune -f + logSuccess "Volumes cleaned." +} + +# Remove container +removeContainer() { + stopOracle + logInfo "Removing Oracle container..." + podman rm $(podman ps -a | grep $ORACLE_CONTAINER | awk '{print $1}') + if [ $? -eq 0 ]; then + logSuccess "Container removed successfully." + else + logError "Failed to remove container." + fi +} + +# Get bash access to container +bashAccess() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening bash shell in container..." + podman exec -it $orclImage /bin/bash +} + +# Get root access to container +rootAccess() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening root shell in container..." + podman exec -it -u 0 $orclImage /bin/bash +} + +# Get SQLPlus nolog access +sqlPlusNolog() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening SQLPlus session (no login)..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; sqlplus /nolog" +} + +# Get SYSDBA access +sysDba() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Opening SQLPlus session as SYSDBA..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; /home/oracle/sqlcl/bin/sql sys/$ORACLE_SYS_PASS@'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=$ORACLE_PDB)))' as sysdba" +} + +# Create user account +createUser() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Creating user account..." + podman exec -it $orclImage bash -c "source /home/oracle/.bashrc; /home/oracle/sqlcl/bin/sql sys/$ORACLE_SYS_PASS@'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=$ORACLE_PDB)))' as sysdba <>/etc/yum.repos.d/mongodb-org-8.0.repo" + + podman exec -i -u 0 $orclImage /usr/bin/yum install -y mongodb-mongosh + logSuccess "MongoDB tools installed successfully." +} + +# Start MongoDB inside the Oracle container +startMongoDB() { + export container_id=$(getContainerId) + + if [ -z "$container_id" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Checking if MongoDB is already running..." + podman exec -i $container_id pgrep mongod >/dev/null 2>&1 + if [ $? -eq 0 ]; then + logInfo "MongoDB is already running in the container." + return 0 + fi + + logInfo "MongoDB is not running. Installing MongoDB..." + + podman exec -i -u 0 $container_id /usr/bin/yum install -y mongodb-org + + logInfo "Configuring MongoDB on port 23456..." + podman exec -i -u 0 $container_id /usr/bin/bash -c " + sed -i 's|bindIp: 127.0.0.1|bindIp: 0.0.0.0|' /etc/mongod.conf + sed -i 's|port: 27017|port: 23456|' /etc/mongod.conf + " + + logInfo "Starting MongoDB instance..." + + # No Replica set config + podman exec -d -u 0 $orclImage /usr/bin/mongod --bind_ip_all --config /etc/mongod.conf + + # Replica set configuration + #podman exec -d -u 0 $container_id /usr/bin/mongod --bind_ip_all --replSet myReplSet --config /etc/mongod.conf + + # get IP address for setting replica set use + # rsIPAddr=$(podman inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' "$container_id") + # echo $rsIPAddr + + # 1 member + # rsConfig="{_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' }]}" + + # 2 members + # rsConfig="{_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' },{ _id: 1, host: 'localhost:23456' }]}" + # echo $rsConfig + + sleep 1 + # start replSet for GG CDC + # podman exec -u 0 $container_id mongosh --port 23456 --eval "rs.initiate(${rsConfig});rs.status();" + + # podman exec -u 0 $container_id mongosh --port 23456 --eval "rs.reconfig({_id: 'myReplSet',members: [{ _id: 0, host: '${rsIPAddr}:23456' }]}, { force: true });rs.status();" + + + # Verify it started + sleep 2 + podman exec -i $container_id pgrep mongod >/dev/null 2>&1 + if [ $? -eq 0 ]; then + logSuccess "MongoDB started successfully." + else + logError "MongoDB failed to start." + return 1 + fi +} + +# Install utilities +installUtils() { + logInfo "Installing useful tools after provisioning container..." + logWarning "Please be patient as this can take time given network latency." + + checkPodman + export container_id=$(getContainerId) + + if [ -z "$container_id" ]; then + logError "Oracle container is not running." + return 1 + fi + + # workaround for ol repo issues + logInfo "Configuring YUM repositories..." + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/touch /etc/yum/vars/ociregion" + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/echo > /etc/yum/vars/ociregion" + + # Add sudo access for oracle user + podman exec -it -u 0 $container_id /bin/bash -c "/usr/bin/echo 'oracle ALL=(ALL) NOPASSWD: ALL' >>/etc/sudoers" + + # Install EPEL repository + logInfo "Installing EPEL repository..." + podman exec -it -u 0 $container_id /usr/bin/rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm + podman exec -it -u 0 $container_id /usr/bin/yum update -y + + # Install required packages + logInfo "Installing required packages..." + podman exec -it -u 0 $container_id /usr/bin/yum install -y sudo which java-17-openjdk wget htop lsof zip unzip rlwrap git python3.12 python3-pip + + # Install pip for Python 3.12 + logInfo "Installing pip for Python 3.12..." + podman exec -it -u 0 $container_id /bin/bash -c "curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py; python3.12 /tmp/get-pip.py" + + # Install Node.js + logInfo "Installing Node.js..." + podman exec -it -u 0 $container_id bash -c "curl -fsSL https://rpm.nodesource.com/setup_20.x | bash - && yum install -y nodejs" + + # Download and install ORDS + logInfo "Downloading and installing ORDS..." + podman exec $container_id /usr/bin/wget -O /home/oracle/ords.zip https://download.oracle.com/otn_software/java/ords/ords-latest.zip + podman exec $container_id /usr/bin/unzip /home/oracle/ords.zip -d /home/oracle/ords/ + + + # Download and install SQLcl + logInfo "Downloading and installing SQLcl..." + podman exec $container_id /usr/bin/wget -O /home/oracle/sqlcl.zip https://download.oracle.com/otn_software/java/sqldeveloper/sqlcl-latest.zip + podman exec $container_id /usr/bin/unzip /home/oracle/sqlcl.zip -d /home/oracle/ + # add alias for sqlcl to be invoked by sqlplus + podman exec -i -u 0 $container_id /bin/bash -c "echo 'alias sqlplus=/home/oracle/sqlcl/bin/sql' >> /home/oracle/.bashrc" + + + # Install MongoDB tools + installMongoTools + + # Install personal tools + logInfo "Installing personal tools..." + podman exec $container_id /usr/bin/wget -O /tmp/PS1.sh https://raw.githubusercontent.com/mattdee/orclDocker/main/PS1.sh + podman exec $container_id /bin/bash /tmp/PS1.sh + podman exec $container_id /usr/bin/wget -O /opt/oracle/product/23ai/dbhomeFree/sqlplus/admin/glogin.sql https://raw.githubusercontent.com/mattdee/orclDocker/main/glogin.sql + + # Set Oracle password + setOrclPwd + + logSuccess "Utilities installation complete!" +} + +# Copy file into container +copyIn() { + checkPodman + export orclRunning=$(getContainerId) + + if [ -z "$orclRunning" ]; then + logError "Oracle container is not running." + return 1 + fi + + read -p "Please enter the ABSOLUTE PATH to the file you want copied: " thePath + read -p "Please enter the FILE NAME you want copied: " theFile + + logInfo "Copying file: $thePath/$theFile into container..." + podman cp $thePath/$theFile $orclRunning:/tmp + + if [ $? -eq 0 ]; then + logSuccess "File copied successfully to /tmp/$theFile in the container." + else + logError "Failed to copy file into container." + fi +} + +# Copy file out of container +copyOut() { + checkPodman + export orclRunning=$(getContainerId) + + if [ -z "$orclRunning" ]; then + logError "Oracle container is not running." + return 1 + fi + + read -p "Please enter the ABSOLUTE PATH in the CONTAINER to the file you want copied to host: " thePath + read -p "Please enter the FILE NAME in the CONTAINER you want copied: " theFile + + logInfo "Copying file: $orclRunning:$thePath/$theFile to host..." + podman cp $orclRunning:$thePath/$theFile /tmp/ + + if [ $? -eq 0 ]; then + logSuccess "File copied successfully to /tmp/$theFile on your host." + else + logError "Failed to copy file from container." + fi +} + +# Setup ORDS +setupORDS() { + checkPodman + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + # Create temp password file + logInfo "Creating temporary password file..." + podman exec -i -u 0 $orclImage /bin/bash -c "echo '$ORACLE_SYS_PASS' > /tmp/orclpwd" + + logInfo "Configuring ORDS..." + + # Create user for ORDS + createUser + + # ORDS silent setup + logInfo "Installing ORDS..." + podman exec -i $orclImage /bin/bash -c "/home/oracle/ords/bin/ords --config /home/oracle/ords_config install --admin-user SYS --db-hostname localhost --db-port 1521 --db-servicename $ORACLE_PDB --log-folder /tmp/ --feature-sdw true --feature-db-api true --feature-rest-enabled-sql true --password-stdin /dev/null 2>&1; sleep 10" + sleep 5 + + # Verify ORDS is running + podman exec $orclImage /bin/bash -c "/usr/bin/ps -ef | grep -i ords" + + logSuccess "ORDS started successfully!" +} + +# Stop ORDS +stopORDS() { + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + logInfo "Stopping ORDS..." + podman exec $orclImage /bin/bash -c "for i in $(ps -ef | grep ords | awk '{print $2}'); do echo $i; kill -9 $i; done" + logSuccess "ORDS stopped successfully!" +} + +# Check MongoDB API +checkMongoAPI() { + # Test MongoDB connections in the container + logInfo "Checking MongoDB API health..." + export orclImage=$(getContainerId) + + if [ -z "$orclImage" ]; then + logError "Oracle container is not running." + return 1 + fi + + # run disableTelemetry() + logInfo "Creating test collection..." + podman exec -it $orclImage bash -c "mongosh --tlsAllowInvalidCertificates 'mongodb://$ORACLE_USER:$ORACLE_PASS@127.0.0.1:27017/$ORACLE_USER?authMechanism=PLAIN&ssl=true&retryWrites=false&loadBalanced=true'< matthew.demarco@oracle.com " + echo -e "${BLUE}===========================================================================${NC}" + exit 0 + ;; + "") + logInfo "No args provided. Starting menu interface..." + ;; + *) + logError "Invalid argument: $1" + logInfo "Run '$0 help' for usage information." + exit 1 + ;; +esac + +# Main menu loop +while true; do + displayMenu + + case $menuChoice in + 1) + startOracle + ;; + 2) + stopOracle + ;; + 3) + bashAccess + ;; + 4) + rootAccess + ;; + 5) + removeContainer + ;; + 6) + installUtils + ;; + 7) + copyIn + ;; + 8) + copyOut + ;; + 9) + cleanVolumes + ;; + 10) + doNothing + ;; + 11) + sqlPlusNolog + ;; + 12) + sqlPlusUser + ;; + 13) + sysDba + ;; + 14) + setupORDS + ;; + 15) + serveORDS + ;; + 16) + checkMongoAPI + ;; + 17) + startMongoDB + ;; + 18) + mongoshORDS + ;; + 19) + mongoshMongoDB + ;; + 20) + runDemoApp + ;; + 21) + addData + ;; + 22) + migrateData + ;; + 23) + listPorts + ;; + *) + badChoice + ;; + esac + + # Reset count after a valid choice + if [[ $menuChoice =~ ^[1-9]|1[0-9]|2[0-2]$ ]]; then + INVALID_COUNT=0 + fi + + # Pause after each operation to view results + if [[ $menuChoice != 10 && $menuChoice =~ ^[1-9]|1[0-9]|2[0-2]$ ]]; then + echo + read -p "Press Enter to continue..." dummy + fi +done \ No newline at end of file diff --git a/mongo-2-oracle/step-by-step-guide.md b/mongo-2-oracle/step-by-step-guide.md new file mode 100644 index 0000000..7a49744 --- /dev/null +++ b/mongo-2-oracle/step-by-step-guide.md @@ -0,0 +1,270 @@ +# MongoDB-2-Oracle Migration: Step-by-step runbook + +This detailed runbook will guide you through the process of using the `orclMongoMigration.sh` script to demonstrate migrating data from MongoDB to Oracle Database. + + +## Prerequisites +Before you begin, ensure your system meets these requirements: +* **Operating System**: macOS or Linux +* **Hardware**: + * At least 8GB RAM (16GB recommended) + * At least 20GB free disk space +* **Internet Connection**: Required for downloading container images and dependencies +* **Terminal Access**: You’ll be working in the command line + +## Getting Started + +### Step 1: Make the Script Executable +1. Open your terminal application +2. Navigate to the directory containing the script: + + ```bash + cd + ``` +3. Make the script executable: + + ```bash + + chmod +x orclMongoMigration.sh + + ``` +### Step 2: Understanding the Script’s Interface +There are two ways to interact with the script: +1. **Interactive Menu**: Run the script without arguments +2. **Command Line Arguments**: Run specific commands directly + +For this runbook, we’ll primarily use the interactive menu for clarity, with command examples where helpful. + +## Container Management + +### Step 3: Starting the Oracle Container +1. Run the script without arguments to access the menu: + + ```bash + + ./orclMongoMigration.sh + + ``` + +2. You’ll see a color-coded menu with several sections +3. Press 1 to start the Oracle container +4. If Podman is not installed, the script will offer to install it for you: + * Type y and press Enter if prompted to install Podman + * The script will install Homebrew (if needed) and then Podman + * This process may take several minutes +5. Choose the Oracle Database version: + * Type 2 and press Enter to select the Full Version (required for MongoDB API) +6. The script will download and start the Oracle container: + * This is a large download (several GB) and may take time depending on your internet connection + * Be patient during the initial download + * After downloading, the container will initialize (this takes several minutes) +7. The script will automatically install utilities in the container: + * Various Linux utilities + * Python 3.12 + * Node.js + * MongoDB tools + * ORDS (Oracle REST Data Services) +8. When complete, you’ll see a list of mapped ports and a success message +9. Press Enter to return to the main menu + +### Step 4: Checking Container Status +1. You can verify the container is running with option 3 (Bash access) +2. This will open a shell inside the container +3. Type exit to return to the script menu + +## Database Setup + +### Step 5: Setting Up ORDS (Oracle REST Data Services) +1. From the main menu, press 14 to set up ORDS +2. The script will: + * Create necessary users + * Configure database settings + * Enable the MongoDB API + * Start the ORDS service +3. Wait for the “ORDS setup complete!” message +4. Press Enter to return to the main menu + +### Step 6: Starting MongoDB +1. From the main menu, press 17 to start MongoDB instance +2. The script will: + * Install MongoDB if needed + * Configure MongoDB to listen on port 23456 + * Start the MongoDB service +3. Wait for the “MongoDB started successfully” message +4. Press Enter to return to the main menu + +## Working with MongoDB + +### Step 7: Connecting to MongoDB +1. From the main menu, press 19 to connect to MongoDB directly +2. This will open the MongoDB shell (mongosh) +3. Try some basic MongoDB commands: + + ```bash + // Show databases + show dbs + + // Create a new database + use testdb + + // Create a collection and insert a document + db.testcollection.insertOne({ name: "Test User", email: "test@example.com" }) + + // Query the collection + db.testcollection.find() + ``` + +4. Type exit to exit the MongoDB shell and return to the script menu + +## Running the Demo Application + +### Step 8: Getting the Demo Application +1. From the main menu, press 20 to run the Registration Demo App +2. The script will: + * Clone the application repository if needed + * Install Node.js dependencies + * Prompt the user as to what datastore to use for the application + * 1 will use MongoDB + * 2 will use the Oracle Database API for MongoDB + * 3 will use Autonomous JSON database (still in development) + * Start the application on port 3000 +3. Access the application in your web browser at: + http://localhost:3000 +4. The demo application allows you to register users +5. Press Ctrl+C in your terminal to stop the application and return to the script + +### Step 9: Adding Demo Data +1. From the main menu, press 21 to add demo data +2. The script will: + * Install required Python libraries + * Set up the geodata directory + * Download necessary shapefiles + * Start an interactive session to run the data generation script +3. When prompted: + * Enter 10 for the number of fake records + * Enter 3 for “No images” option (simpler) +4. The script will generate fake data with geospatial coordinates within the shapefiles +5. Press Enter to return to the main menu + +## Data Migration Process + +### Step 10: Migrating Data from MongoDB to Oracle +1. From the main menu, press 22 to start the migration process +2. Read the migration information displayed +3. Press Enter to start the interactive migration tool +4. In the migration tool menu: + * Choose option 3 for “Both Export & Import” +5. For MongoDB export (source) settings: + * Enter localhost:23456/test as your MongoDB endpoint + * Leave the authentication fields blank (press Enter) + * Type no for SSL/TLS connection + * Press Enter to export all collections (or specify a collection) + * Press Enter to use default parallel collections + * Enter /tmp/moveit as the export location +6. For Oracle MongoDB API (target) settings: + * Enter matt as the username + * Enter matt as the password + * Enter localhost as the hostname + * Enter matt as the schema name + * Enter /tmp/moveit as the import location +7. The script will: + * Export data from MongoDB to files + * Import data into Oracle Database via the MongoDB API + * Show progress in real-time +8. When the migration completes, you’ll be asked if you want to verify the migration + * Type y and press Enter to open SQL*Plus for verification +9. In SQL*Plus, run verification commands: + + ```bash + -- List tables + SELECT table_name FROM user_tables; + + -- Count records in the migrated collection + SELECT COUNT(*) FROM testcollection; + + -- View sample data + SELECT * FROM testcollection WHERE ROWNUM <= 5; + + -- Exit SQL*Plus + EXIT + ``` + +10. Press Enter to return to the main + +## Verification and Troubleshooting + +### Step 11: Checking MongoDB API Connection +1. From the main menu, press 16 to check MongoDB API connection +2. The script will: + * Create a test collection + * Insert a test document + * Read back the test document +3. If successful, you’ll see the test document data displayed +4. Press Enter to return to the main menu + +## Step 12: Connecting to Oracle via MongoDB API +1. From the main menu, press 18 to connect to Oracle via MongoDB API (mongosh ORDS) +2. This opens a MongoDB shell connected to Oracle’s MongoDB API +3. Try some commands: + + ```bash + // Show collections + show collections + + // Query the migrated collection + db.testcollection.find() + + // Create a new collection in Oracle via MongoDB API + db.newcollection.insertOne({ name: "Oracle API Test", value: 42 }) + + // Verify it was created + db.newcollection.find() + ``` + +4. Type exit to exit the shell and return to the script menu + +## Shutting Down + +### Step 13: Stopping the Container +1. From the main menu, press 2 to stop the Oracle container +2. The script will stop all running Oracle containers +3. Press Enter to return to the main menu +4. Press 10 to exit the script + +## Command Reference +For advanced users or automation, you can run specific commands directly: + +### Container Management + + ```bash + ./orclMongoMigration.sh start # Start Oracle container + ./orclMongoMigration.sh stop # Stop Oracle container + ./orclMongoMigration.sh bash # Bash access to container + ./orclMongoMigration.sh root # Root access to container + ``` + +### Database Operations + + ``` + ./orclMongoMigration.sh sqluser # SQL*Plus user connection + ./orclMongoMigration.sh sqlsys # SQL*Plus SYSDBA connection + ./orclMongoMigration.sh ords # Start ORDS service + MongoDB Operations + ./orclMongoMigration.sh mongodb # Start MongoDB instance + ./orclMongoMigration.sh mongo # Connect to MongoDB directly + ./orclMongoMigration.sh mongoords # Connect to MongoDB via ORDS + ``` + +### Application and Migration + + ```bash + ./orclMongoMigration.sh demoapp # Run Registration Demo App + ./orclMongoMigration.sh demodata # Add demo data + ./orclMongoMigration.sh migrate # Migrate data + Help and Information + ./orclMongoMigration.sh help # Show all available commands + ``` + +## Summary +________________________________________ +This runbook covers the basic workflow for demonstrating MongoDB to Oracle migration. You can explore additional features and options as you become more familiar with the script. From bbae1d8c18cc277946672a81fe8e095a45bcdad1 Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Thu, 7 Aug 2025 11:43:16 -0700 Subject: [PATCH 2/6] Adding resource links --- mongo-2-oracle/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md index 03ef019..a4ce800 100644 --- a/mongo-2-oracle/README.md +++ b/mongo-2-oracle/README.md @@ -43,7 +43,7 @@ Demo Application ```bash - wget + wget https://github.com/oracle-devrel/devo.tutorials/blob/mongo-2-oracle-migration/mongo-2-oracle/src/orclMongoMigration.sh ``` @@ -68,6 +68,8 @@ Demo Application * **MongoDB Operations:** - Start MongoDB instance - mongosh to ORDS or MongoDB * **Application & Migration:** - Run registration demo app - Add demo data - Migrate data from MongoDB to Oracle +4. For additional details, please visit the [step-by-step guide](step-by-step-guide.md). + ## Available Commandline Arguments * Container Management: From bd5d102ea5bf3f6bafb8f12f5b8330c487ba7e59 Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Thu, 7 Aug 2025 11:44:01 -0700 Subject: [PATCH 3/6] Adjusting format --- mongo-2-oracle/README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md index a4ce800..fd05540 100644 --- a/mongo-2-oracle/README.md +++ b/mongo-2-oracle/README.md @@ -42,9 +42,7 @@ Demo Application 1. Download the script ```bash - wget https://github.com/oracle-devrel/devo.tutorials/blob/mongo-2-oracle-migration/mongo-2-oracle/src/orclMongoMigration.sh - ``` 2. Change permissions to make the script executable From 9709bf78a194965dbb342416bbafecae7a89f58e Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Thu, 7 Aug 2025 11:45:54 -0700 Subject: [PATCH 4/6] Formatting adjustments --- mongo-2-oracle/README.md | 12 ------------ mongo-2-oracle/step-by-step-guide.md | 4 ---- 2 files changed, 16 deletions(-) diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md index fd05540..5e3151f 100644 --- a/mongo-2-oracle/README.md +++ b/mongo-2-oracle/README.md @@ -48,17 +48,13 @@ Demo Application 2. Change permissions to make the script executable ```bash - chmod +x orclMongoMigration.sh - ``` 3. Run the script without arguments to display a user-friendly menu with the following sections: ```bash - ./orclMongoMigration.sh - ``` * **Container Management:** - Start/stop Oracle container - Bash and root access - Install utilities and manage volumes - Copy files in and out @@ -118,27 +114,19 @@ Here’s a typical workflow for demonstrating MongoDB to Oracle migration: 1. Start the container: ```bash - ./orclMongoMigration.sh start - ``` 2. Start MongoDB instance: ```bash - ./orclMongoMigration.sh mongodb - ``` 3. Generate demo data: ```bash - ./orclMongoMigration.sh demodata - ``` 4. Perform the migration: ```bash - ./orclMongoMigration.sh migrate - ``` 5. Verify the migrated data in Oracle: (The script will offer to connect you to SQL*Plus after migration) diff --git a/mongo-2-oracle/step-by-step-guide.md b/mongo-2-oracle/step-by-step-guide.md index 7a49744..11ef972 100644 --- a/mongo-2-oracle/step-by-step-guide.md +++ b/mongo-2-oracle/step-by-step-guide.md @@ -24,9 +24,7 @@ Before you begin, ensure your system meets these requirements: 3. Make the script executable: ```bash - chmod +x orclMongoMigration.sh - ``` ### Step 2: Understanding the Script’s Interface There are two ways to interact with the script: @@ -41,9 +39,7 @@ For this runbook, we’ll primarily use the interactive menu for clarity, with c 1. Run the script without arguments to access the menu: ```bash - ./orclMongoMigration.sh - ``` 2. You’ll see a color-coded menu with several sections From 0f491a18e76f690ad8987c4e6345ce52627c17fb Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Mon, 25 Aug 2025 15:48:55 -0700 Subject: [PATCH 5/6] Update instructions --- mongo-2-oracle/src/orclMongoMigration.sh | 8 ++++---- mongo-2-oracle/step-by-step-guide.md | 22 ++++++++++++++++++++-- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/mongo-2-oracle/src/orclMongoMigration.sh b/mongo-2-oracle/src/orclMongoMigration.sh index 57a8851..99e1448 100644 --- a/mongo-2-oracle/src/orclMongoMigration.sh +++ b/mongo-2-oracle/src/orclMongoMigration.sh @@ -53,10 +53,10 @@ # SOFTWARE. # Global variables -ORACLE_USER="matt" -ORACLE_PASS="matt" -ORACLE_SYS_PASS="Oradoc_db1" -ORACLE_CONTAINER="MoMoMatt" +ORACLE_USER="<--->" +ORACLE_PASS="<--->" +ORACLE_SYS_PASS="<--->" +ORACLE_CONTAINER="<--->" ORACLE_PDB="FREEPDB1" NETWORK_NAME="demonet" #set network CONTAINER_PORT_MAP="-p 1521:1521 -p 3000:3000 -p 5902:5902 -p 5500:5500 -p 8000:8000 -p 8080:8080 -p 8443:8443 -p 27017:27017 -p 23456:23456" diff --git a/mongo-2-oracle/step-by-step-guide.md b/mongo-2-oracle/step-by-step-guide.md index 11ef972..03dde7a 100644 --- a/mongo-2-oracle/step-by-step-guide.md +++ b/mongo-2-oracle/step-by-step-guide.md @@ -26,12 +26,30 @@ Before you begin, ensure your system meets these requirements: ```bash chmod +x orclMongoMigration.sh ``` -### Step 2: Understanding the Script’s Interface +### Step 2: Understanding the Script’s Interface and global variables There are two ways to interact with the script: 1. **Interactive Menu**: Run the script without arguments 2. **Command Line Arguments**: Run specific commands directly -For this runbook, we’ll primarily use the interactive menu for clarity, with command examples where helpful. + >NOTE: For this runbook, we’ll primarily use the interactive menu for clarity, with command examples where helpful. + +3. Before running the script, it is important to update the global variables to match your environment. Edit the following items: + + ```bash + # Global variables + ORACLE_USER="" + ORACLE_PASS="" + ORACLE_SYS_PASS="" + ORACLE_CONTAINER="" + ORACLE_PDB="FREEPDB1" + NETWORK_NAME="demonet" #set network + // Do not change the settings below this line + CONTAINER_PORT_MAP="-p 1521:1521 -p 3000:3000 -p 5902:5902 -p 5500:5500 -p 8000:8000 -p 8080:8080 -p 8443:8443 -p 27017:27017 -p 23456:23456" + MAX_INVALID=3 + INVALID_COUNT=0 + ``` + +4. Save the script and mvoe to the next step. ## Container Management From 9e13e5ca0b0a92a5bb375ac41f2bd094b665a9e4 Mon Sep 17 00:00:00 2001 From: Eli Schilling Date: Mon, 25 Aug 2025 15:59:24 -0700 Subject: [PATCH 6/6] Update readme.md --- mongo-2-oracle/README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mongo-2-oracle/README.md b/mongo-2-oracle/README.md index 5e3151f..e83f7f3 100644 --- a/mongo-2-oracle/README.md +++ b/mongo-2-oracle/README.md @@ -51,7 +51,11 @@ Demo Application chmod +x orclMongoMigration.sh ``` -3. Run the script without arguments to display a user-friendly menu with the following sections: +3. Set the global variables within the script file. + + >NOTE: See [the step by step guide](step-by-step-guide.md) for details on setting global variables. + +4. Run the script without arguments to display a user-friendly menu with the following sections: ```bash ./orclMongoMigration.sh @@ -62,7 +66,7 @@ Demo Application * **MongoDB Operations:** - Start MongoDB instance - mongosh to ORDS or MongoDB * **Application & Migration:** - Run registration demo app - Add demo data - Migrate data from MongoDB to Oracle -4. For additional details, please visit the [step-by-step guide](step-by-step-guide.md). +5. For a comprehensive walkthru, please visit the [step-by-step guide](step-by-step-guide.md). ## Available Commandline Arguments