Updating the list?

[RabbitThatIsPink]

I see in /Dev, theres a few scrapers. How would one use them to update the list?

[rainlizard]

There's two types of lists:

• The taglists that describe each image: taglists-general.txt, taglists-questionable.txt, taglists-sensitive.txt, taglists-explicit.txt, etc.

• The list of individual tags (with categories as prefix): categorized_tags.txt

---

Here's how to use the scraper for the taglists:

1. Create an account on https://danbooru.donmai.us/
2. Go to https://danbooru.donmai.us/profile
3. At the bottom is API Key, click View
4. Add an API key
5. Copy the random string of letters you're given
6. Edit the /dev/taglist-scraper/danbooru-taglist-scraper.py file
7. On line 95 put in your danbooru username, on line 96 put in your API key.
8. Make sure Python is installed
9. Run the danbooru-taglist-scraper.py file (or you can also run danbooru-taglist-scraper.bat)
10. Wait while text files (general.txt, questionable.txt, sensitive.txt, explicit.txt) are slowly filled line by line inside the output_scraped folder.

Taglists are very easy to update, I stopped at 100,000 items in each text file since it could possibly start slowing processing down if we make them too large, and also github doesn't like files larger than 100MB each, so you may need separate hosting like huggingface to store those datasets.

---

But if you're talking about the categorized_tags.txt file, that's where things get a little tricky. Though you can easily run /dev/tag-scraper/tag-scraper.py without a danbooru account or API key and get a list of tags - they're outputted to the tag_lists folder. But I had to use a fair bit of manual processing to handle categorization and stuff. I can elaborate on this if you want.

[RabbitThatIsPink]

I see. So this just updates the list if it finds a new unique entry? If using the regular taglist-scraper?

[rainlizard]

Assuming you're talking about /taglist-scraper/ and not /tag-scraper/ (it's a bit confusing), then yeah, it just adds more and more lines to the text files in /output_scraped/. You can close and restart the scraper at will, without anything breaking.

So if you wanted, what you could do is copy the /lists/taglists-general.txt file to /dev/taglist-scraper/output_scraped/general.txt (rename it from taglists-general.txt to general.txt) and do the same for the 3 other taglist files, then run the scraper and more entries will be added to those 100,000 entries. Their order in the file doesn't matter.

This scraper is a little random in what it scrapes, it tries to prioritize images that have high scores, however it only looks for a little while before lowering the score threshold and then scraping things with lower scores too. It avoids any duplicate entries, and tries to fill the file that has the least number of entries (if general.txt has less entries than questionable.txt, then general.txt gets prioritized).

Quick screenshot of the scraper in action:

(It's saying "API returned 4 posts" but it can return up to 200 at a time, it just depends.)