One Stream Model Training with RHM Dataset

This repository is dedicated to training one stream models, specifically C3D, SlowFast, R2+1D, and R3D, using the RHM (Robotics Human Monitoring) dataset. It's optimized for advanced video analysis tasks in robotics through single-stream video processing.

Note

This project focuses on single-stream 3D CNN models for human activity recognition in multi-view robotics environments.

🌐 Project Page: https://bamorovat.com/projects/rhm-dataset.html

Project Structure

RHM_OneView/
├── src/                          # Source code
│   ├── models/                   # Model architectures
│   │   ├── C3D_model.py         # C3D model implementation
│   │   ├── R2Plus1D_model.py    # R2+1D model implementation
│   │   ├── R3D_model.py         # R3D model implementation
│   │   └── Slow_Fast_model.py   # SlowFast model implementation
│   ├── data/                     # Data loading and processing
│   │   └── dataloader.py        # RHM dataset loader
│   └── utils/                    # Utility functions
│       ├── PretrainModelPath.py # Pretrained model path management
│       ├── training_utils.py    # Training utilities
│       └── visualization.py     # Visualization functions
├── configs/                      # Configuration files
│   └── config.py                # Model and training configurations
├── scripts/                      # Training and evaluation scripts
│   ├── train.py                 # Main training script
│   └── evaluate.py              # Model evaluation script
├── data/                        # Dataset split files
│   ├── trainlist.txt           # Training data splits
│   ├── vallist.txt             # Validation data splits
│   └── testlist.txt            # Test data splits
├── pretrained_models/           # Directory for pretrained models
├── main.py                      # Main entry point
├── requirements.txt             # Python dependencies
└── README.md                   # This file

Files Description

• main.py: Main entry point with command-line interface for easy training
• src/models/: Contains the model architectures for C3D, SlowFast, R2+1D, and R3D models
• src/data/dataloader.py: Implements efficient data loading logic for the RHM dataset
• src/utils/: Utility functions for training, visualization, and model path management
• configs/config.py: Centralized configuration with model-specific parameters
• scripts/train.py: Core training pipeline implementation
• scripts/evaluate.py: Model evaluation with metrics and visualization
• data/: Contains dataset split files for training, validation, and testing

Requirements

Important

Ensure all dependencies are properly installed before running the training pipeline.

System Requirements

• Python 3.8+
• CUDA-compatible GPU (recommended)
• 8GB+ RAM
• 50GB+ storage for dataset and models

Dependencies

• PyTorch >= 1.9.0
• TorchVision >= 0.10.0
• OpenCV >= 4.5.0
• NumPy >= 1.21.0
• TensorBoardX >= 2.4
• Matplotlib >= 3.4.0
• Seaborn >= 0.11.0
• Pandas >= 1.3.0
• Scikit-learn >= 1.0.0

Installation

1. Clone the repository:

   git clone https://github.com/Bamorovat/RHM_OneView.git
   cd RHM_OneView

2. Install dependencies:

   pip install -r requirements.txt

3. Configure paths in configs/config.py:

   BASE_CONFIG = {
       'dataset_path': '/path/to/your/RHM_dataset',  # Update this path
       'save_path': '/path/to/output/directory',     # Update this path
       # ... other configurations
   }

4. Download pretrained models (see Pretrained Models section)

Tip

Create a virtual environment before installation to avoid dependency conflicts:

python -m venv rhm_env
source rhm_env/bin/activate  # On Windows: rhm_env\Scripts\activate

Usage

Quick Start

Tip

Start with the C3D model and OmniView for initial experiments, as it provides good balance between performance and computational requirements.

1. Configure your experiment in configs/config.py:

• Choose model: C3D, SlowFast, R2Plus1D, or R3D
• Set view: FrontView, BackView, OmniView, or RobotView
• Set view_status: Normal, OpticalFlow, MotionHistoryImages, etc.

2. Start training:

# Using main entry point with command-line arguments
python main.py --model C3D --view OmniView --batch_size 32 --lr 1e-4

# Or run training script directly
cd scripts
python train.py

3. Monitor training:

tensorboard --logdir logs/

Model Evaluation

After training, evaluate your model:

cd scripts
python evaluate.py --model C3D --checkpoint /path/to/checkpoint.pth --save_results ./results

Warning

Ensure your checkpoint path is correct and the model architecture matches the saved checkpoint.

Available Models

Model	Description	Memory Usage	Training Time
C3D	Standard 3D CNN	Medium	Fast
SlowFast	Dual-pathway network	High	Slow
R2Plus1D	2+1D convolution	Medium	Medium
R3D	3D ResNet	Medium	Medium

Pretrained Models

Important

Download pretrained models to improve training convergence and final performance.

Required model files in pretrained_models/ directory:

Model	Filename	Source
C3D	c3d-pretrained.pth	C3D Official
R3D	r3d_18-b3b3357e.pth	PyTorch Model Zoo
R2+1D	r2plus1d_18-91a641e6.pth	PyTorch Model Zoo
SlowFast	SLOWFAST_4x16_R50.pkl	SlowFast Official

Download commands:

mkdir -p pretrained_models
cd pretrained_models

# Download from PyTorch Hub (R3D and R2+1D)
python -c "import torch; torch.hub.load('pytorch/vision', 'r3d_18', pretrained=True)"
python -c "import torch; torch.hub.load('pytorch/vision', 'r2plus1d_18', pretrained=True)"

Note

C3D and SlowFast models need to be downloaded manually from their respective repositories.

Dataset Preparation

Warning

The RHM dataset requires preprocessing before training. Raw video files must be converted to frame sequences.

Steps to prepare the dataset:

1. Extract frames from videos:

   git clone https://github.com/Bamorovat/RHM_Frame_Feature_Extractor.git
   # Follow the extraction repository instructions

2. Organize dataset structure:

   RHM_dataset/
   ├── FrontView_ExtractedFrames/
   ├── BackView_ExtractedFrames/
   ├── OmniView_ExtractedFrames/
   ├── RobotView_ExtractedFrames/
   ├── trainlist.txt
   ├── vallist.txt
   └── testlist.txt
   

Tip

Copy the split files from the data/ directory to your dataset root directory for compatibility.

Dataset Access

Important

The RHM dataset requires permission for access and use.

To obtain the RHM dataset:

• Contact: Patrick Holthaus at p.holthaus@herts.ac.uk
• Website: Robot House Project
• Research Purpose: Clearly state your research objectives when requesting access

Citation

Important

If you use the RHM dataset or this code in your research, please cite the relevant papers below.

Primary Dataset Paper

Mohammad Hossein Bamorovat Abadi, Mohamad Reza Shahabian Alashti, Patrick Holthaus, Catherine Menon, and Farshid Amirabdollahian. "RHM: Robot House Multi-view Human Activity Recognition Dataset." ACHI 2023, Venice, Italy, IARIA.

BibTeX:

@inproceedings{bamorovat2023rhm,
  title={RHM: Robot House Multi-view Human Activity Recognition Dataset},
  author={Bamorovat Abadi, Mohammad Hossein and Shahabian Alashti, Mohamad Reza and Holthaus, Patrick and Menon, Catherine and Amirabdollahian, Farshid},
  booktitle={ACHI 2023: The Sixteenth International Conference on Advances in Computer-Human Interactions},
  year={2023},
  organization={IARIA},
  address={Venice, Italy}
}

Additional Related Publications

Mohammad Hossein Bamorovat Abadi, Mohamad Reza Shahabian Alashti, Patrick Holthaus, Catherine Menon, and Farshid Amirabdollahian. "Robot house human activity recognition dataset." 4th UK-RAS Conference: Robotics at Home (UKRAS21), 19–20. Hatfield, UK, 2021.

BibTeX:

@inproceedings{bamorovat2021robot,
  title={Robot house human activity recognition dataset},
  author={Bamorovat Abadi, Mohammad Hossein and Shahabian Alashti, Mohamad Reza and Holthaus, Patrick and Menon, Catherine and Amirabdollahian, Farshid},
  booktitle={4th UK-RAS Conference: Robotics at Home (UKRAS21)},
  pages={19--20},
  year={2021},
  address={Hatfield, UK}
}

Mohammad Hossein Bamorovat Abadi, Mohamad Reza Shahabian Alashti, Patrick Holthaus, Catherine Menon, and Farshid Amirabdollahian. "Robotic Vision and Multi-View Synergy: Action and activity recognition in assisted living scenarios." BioRob 2024, Heidelberg, Germany, IEEE.

BibTeX:

@inproceedings{bamorovat2024robotic,
  title={Robotic Vision and Multi-View Synergy: Action and activity recognition in assisted living scenarios},
  author={Bamorovat Abadi, Mohammad Hossein and Shahabian Alashti, Mohamad Reza and Holthaus, Patrick and Menon, Catherine and Amirabdollahian, Farshid},
  booktitle={2024 10th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob)},
  year={2024},
  organization={IEEE},
  address={Heidelberg, Germany}
}

📄 Paper Links:

• ACHI 2023 Paper
• Project Page

Development

Adding New Models

Tip

Follow the existing model structure when adding new architectures for consistency.

Steps to add a new model:

1. Create model class in src/models/your_model.py
2. Add model configuration in configs/config.py
3. Update imports in training and evaluation scripts
4. Test model creation before training

Customizing Data Loading

Modify src/data/dataloader.py to:

• Add new preprocessing methods
• Support different input formats
• Implement custom augmentations
• Handle different video resolutions

Troubleshooting

Common Issues

Caution

Always run scripts from the project root directory to avoid import errors.

Issue	Solution
Import Errors	Run from project root or use main.py entry point
Path Issues	Update dataset_path and save_path in configs/config.py
Memory Issues	Reduce batch_size, especially for SlowFast model
CUDA Issues	Verify PyTorch CUDA version matches your system
Dataset Not Found	Check dataset path and file structure

Performance Optimization

Tip

These optimizations can significantly speed up training:

• Mixed Precision: Add --fp16 flag for faster training
• Data Loading: Increase num_workers based on CPU cores
• Batch Size: Use largest batch size that fits in GPU memory
• Monitoring: Use TensorBoard for real-time training metrics

# Example optimized training command
python main.py --model C3D --batch_size 64 --num_workers 8

License

This project is licensed under the GNU General Public License (GPL) v3.