content similarity detection / deduplication

[geeknik]

"Dynamic Scope" integration, cuts back on data usage while crawling by utilizing a TF-IDF machine learning model to discard pages which might be too similar to pages already crawled. 👍🏻

For example, when running katana -d 1 -u https://www.ibm.com/ -j -o ibm.json, the ibm.json is about 11MB. Now when running with Dynamic Scope which adds -uds to the command line, drops the output of ibm.json to about 3.4MB.

Crawl Fast, Crawl Smart. 🚀

   -sdd, -similarity-deduplication enable content similarity detection to avoid crawling similar pages
   -st, -similarity-threshold string similarity threshold for content deduplication (0.0-1.0, default 0.1) (default "0.1")

[Mzack9999]

@geeknik very interesting approach, I'm going to review this soon and compare it with BM25

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[ehsandeep]

• Refactored the code to integrate with the existing engine as an optional feature instead of creating a new engine.
• Added documentation to assist with understanding and testing the feature:
• Renamed options to better reflect the feature and added support for customizing the similarity threshold.

Options:

   -sdd, -similarity-deduplication   Enable content similarity detection to avoid crawling similar pages
   -st,  -similarity-threshold       Set similarity threshold for content deduplication (range: 0.0–1.0, default: 0.1)

[ghost]

When will this feature be available online?