LinkedIn Post Scraper & Email Generator

A Python script to scrape details from LinkedIn job posts and automatically generate personalized application emails using OpenAI.

Features

• Scrapes LinkedIn post content and metadata
• Extracts author information
• Identifies job postings and extracts job details (title, experience, technologies, remote status)
• Extracts hashtags from posts
• Captures engagement metrics (likes, comments, shares)
• Automatically generates personalized job application emails using OpenAI
• Intelligently selects the best resume (Backend or FullStack) based on job requirements
• Saves data to JSON file
• Handles dynamic content loading with Selenium
• Uses Chromium browser with existing profile to maintain login session

Requirements

• Python 3.7+
• Chromium browser (or Chrome)
• ChromeDriver (automatically managed if using webdriver-manager)
• OpenAI API key (for email generation)
• Resume files: Backend Resume.pdf (or .docx) and FullStack Resume.pdf (or .docx)

Installation

1. Install Python dependencies:

pip install -r requirements.txt

2. (Optional) Install webdriver-manager for automatic ChromeDriver management:

pip install webdriver-manager

If you install webdriver-manager, you can modify the script to use it automatically.

Usage

Basic Usage

python3 apply.py

The script will prompt you for the LinkedIn post URL.

With OpenAI API Key (for email generation)

1. Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY='your-api-key-here'

2. Run the script:

python3 apply.py

3. The script will:
   • Scrape the LinkedIn post
   • Automatically select the best resume (Backend or FullStack) based on job requirements
   • Generate a personalized email using OpenAI
   • Save the email to generated_email.txt and generated_email.json

Headless Mode (no browser window)

python3 apply.py --headless

Using Existing Chromium Session

Option 1: Close Chromium and use profile (Recommended)

# 1. Close all Chromium windows
# 2. Run the script
python3 apply.py

Option 2: Use remote debugging (keep Chromium open)

# 1. Start Chromium with remote debugging:
chromium --remote-debugging-port=9222

# 2. Run the script:
python3 apply.py --remote-port 9222

Example

export OPENAI_API_KEY='sk-...'
python3 apply.py
# Enter URL when prompted

Output

The script will:

1. Display scraped data in the console
2. Save the data to linkedin_post_data.json
3. If it's a job post and OpenAI API key is set:
   • Automatically select the best resume (Backend or FullStack)
   • Generate a personalized email
   • Save email to generated_email.txt and generated_email.json

Output Structure

{
  "url": "LinkedIn post URL",
  "post_id": "Post ID extracted from URL",
  "author": {
    "name": "Author name"
  },
  "content": "Full post content",
  "job_details": {
    "hiring": true/false,
    "job_title": "Job title if detected",
    "experience_required": "Years of experience",
    "location": "Location if mentioned",
    "remote": true/false,
    "technologies": ["List of technologies"],
    "requirements": []
  },
  "engagement": {
    "likes": "Number of likes",
    "comments": "Number of comments",
    "shares": "Number of shares"
  },
  "comments": ["Array of comment texts"],
  "hashtags": ["#hashtag1", "#hashtag2"],
  "timestamp": "Post timestamp"
}

Email Generation Features

The email generator uses OpenAI GPT-4o-mini to create personalized job application emails:

• Intelligent Resume Selection: Automatically chooses between Backend Resume and FullStack Resume based on job requirements
• Personalized Content: Matches your skills and experience to the job requirements
• Professional Tone: Generates professional, compelling emails that highlight your fit for the role
• Specific Details: References specific technologies, responsibilities, and requirements from the job post
• Ready to Send: Outputs complete email with subject, greeting, body, and closing

Resume Selection Logic

• FullStack Resume: Used when job requires frontend technologies (React, Angular, Vue, JavaScript, TypeScript)
• Backend Resume: Used for backend-focused positions (.NET, ASP.NET, C#, SQL Server, APIs)

Important Notes

1. OpenAI API Key: Required for email generation. Get one from https://platform.openai.com/api-keys

   • Set as environment variable: export OPENAI_API_KEY='your-key'
   • Or enter when prompted by the script

2. Resume Files: Ensure Backend Resume.pdf (or .docx) and FullStack Resume.pdf (or .docx) are in the same directory

3. Authentication: LinkedIn may require you to sign in manually. The browser window will open, and you may need to sign in to your LinkedIn account.

4. Rate Limiting: Be respectful of LinkedIn's servers. Don't scrape too many posts in quick succession.

5. Terms of Service: Make sure your usage complies with LinkedIn's Terms of Service.

6. ChromeDriver: If you encounter ChromeDriver issues, you can:

   • Install it manually and add to PATH
   • Use webdriver-manager: pip install webdriver-manager
   • Update the script to use webdriver-manager (see example below)

Troubleshooting

ChromeDriver Issues

If you get ChromeDriver errors, try:

pip install webdriver-manager

Then modify the script to use webdriver-manager:

from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# In setup_driver method, replace:
# self.driver = webdriver.Chrome(options=chrome_options)
# with:
service = Service(ChromeDriverManager().install())
self.driver = webdriver.Chrome(service=service, options=chrome_options)

Page Not Loading

• LinkedIn may require authentication - sign in when the browser opens
• Some posts may be private or restricted
• LinkedIn's structure may change, requiring selector updates

Empty Results

• The post may require login to view
• LinkedIn's HTML structure may have changed
• The post may be loading dynamically - the script waits 3 seconds, but you may need to increase wait times

Email Sending

The script can automatically send generated emails via Gmail SMTP.

Setting Up Gmail App Password

Important: Gmail requires an App Password (not your regular password) for SMTP access.

Step 1: Enable 2-Step Verification

1. Go to Google Account Security
2. Under "How you sign in to Google", click 2-Step Verification
3. Follow the prompts to enable it (requires phone number)

Step 2: Generate App Password

1. Go directly to: App Passwords
   • Or: Google Account > Security > 2-Step Verification > App passwords
2. Sign in if prompted
3. Select app: Mail
4. Select device: Other (Custom name)
   • Enter: "LinkedIn Email Sender"
5. Click Generate
6. Copy the 16-character password (format: xxxx xxxx xxxx xxxx)
7. Save it securely - you won't see it again!

Step 3: Use in Script

• When the script asks for your Gmail App Password, paste the 16-character password
• Spaces are optional (you can include or remove them)

Email Sending Process

1. After email generation, the script will ask: "Would you like to send this email now?"
2. Enter yes or y
3. Enter your Gmail address
4. Enter your Gmail App Password (the 16-character one from Step 2)
5. The email will be sent automatically!

Troubleshooting Email Sending

"Authentication failed" error:

• Make sure you're using an App Password, not your regular Gmail password
• Verify 2-Step Verification is enabled
• Check that you copied the full 16-character password correctly

"SMTP error" or connection issues:

• Check your internet connection
• Verify Gmail SMTP is not blocked by firewall
• Try again - Gmail may have temporary rate limits

License

This script is provided as-is for educational purposes. Use responsibly and in accordance with LinkedIn's Terms of Service.