Improve Graph/Chart Response Time #6
Description
Okay, this one is important. When you have a very busy log file you can find that it is in excess of 2 million+ rows for a 24 hour time period. This is hard to load properly. The queries don't really load all 2 million+ rows, they just have to hit that many rows to aggregate them into count piles; per timestamp. I have thought about a couple of ways to improve this, I have tried to load this but it maxes out the Heap space and takes FOREVER to return. It is definitely Neo4j, but how to solve it? I have thought about trying to page the results and do an incremental build, which is doable - but we may face the same problem only magnified by the pagination process. This is because Neo4j still needs to create a total count of the data so still has to iterate over all nodes.
Another idea I had was to further limit the result set, so try and round the timestamp in a Neo4j query. Not sure if this is even possible, if not it should be!, so we need to look into it. This would provide the best performance we can get, if we did this we could also remove the additional logic that does the grouping now.
I prefer to send all data at once to the server, if possible, if not then we should probably look at loading the data incrementally from the client side. This is not my top choice and should be the last option. I can see how we could parse the timestamps and do some date manipulation and only send predetermined allotments of time. We then incrementally build the graph.
If anyone has any other ideas, please feel free to pipe in. Performance tuning Neo4j is also be on the table - maybe not use an embedded version and go for the server version. It may perform better due to not having to deal with the web app overhead. This should be looked at as well - mainly for larger operations probably.