Some Performance improvement ideas

[silentdot]

Hey,

first of all thanks for the fantastic tool, have been using it since a year and helped me a lot :).
I have some ideas to speed up the scrapping process:

• Have a cache for matched songs, like beatport release title and spotify song id.
• If possible multiple threads, maybe one per genre/playlist.

Let me know what you think.

Kind regards
Chris

[rootshellz]

Hi Chris,

Have you pulled the code recently? About a month ago, I implemented some caching related to checking the existing playlists: a844402. This significantly speeds up the tool, you should see if this helps in your situation.

I will look in to some further caching and performance improvements. Your idea of caching matched songs across runs is worth looking in to, but I think a better approach might be to simply check the existing playlist for the genre first, before matching songs, and then not trying to find / match songs that are already in the playlist. (e.g. The first check for a match can be against the existing songs in the playlist.)

The idea of multithreading is also good, however, a major issue I see is random timeouts from the Spotify API. These currently plague my runs of the tool and I imagine other users are impacted too. I'm not sure if this is related to Spotify rate limits or filtering of some sort, unexpected temporal issues on the Spotify side, or some other networking issue. I did notice that the caching fix mentioned above seemed to help this, although it might just be a case of seeing fewer timeouts given far fewer API calls. I need to troubleshoot this further before speeding things up, otherwise I imagine it will just speed up the occurrences of the errors.