Update update_record to call async task for boost_mesage. Add queue for receiving augment processes.#215
Conversation
…or receiving augment processes.
tjacovich
requested review from
Thomas-S-Allen,
mugdhapolimera and
seasidesparrow
|
@mugdhapolimera and @Thomas-S-Allen We are going to be transitioning all the augment processes to use the queue OUTPUT_TASKNAME = "adsmp.tasks.task_update_record"becomes OUTPUT_TASKNAME = "adsmp.tasks.task_augment_record" |
|
@tjacovich Just a question: do you want to keep the machinery to update these via the update-records queue (in task_update_record) as well as in tasks_augment_record? I don't see the clauses removed from the former task |
I would like to, at least in the short term so we don't end up with a bunch of lost augments in the event a version of the code that still uses |
|
We also need to add OUTPUT_QUEUE = "augment-record"
FORWARD_MSG_DICT = [ \
{'OUTPUT_PIPELINE': 'default', \
'OUTPUT_CELERY_BROKER': OUTPUT_CELERY_BROKER,\
'OUTPUT_QUEUE' : OUTPUT_QUEUE, \
'OUTPUT_TASKNAME': OUTPUT_TASKNAME}, \
]To the |
…or receiving augment processes. (#215) * Update update_record to call async task for boost_mesage. Add queue for receiving augment processes. * Broke boost call out of update storage to remove circular import. * removed circular import. * Fix args in task_boost_request.
* adding lockfile scripts to long tasks (#213) * adding aws to requirements (#214) * adding aws to requirements * changing aws version * changing aws version 2 * changing aws version 3 * Added user-defined fields for scix id generation (#212) * Added user-defined fields for scix id generation * updated test cases * new version of scixpipelineutils * updating test scixids * updating test scixids * updating test scixids #3 * modified test cases to have at least title in bib_data * Update requirements.txt Co-authored-by: Taylor Jacovich <tjacovich@cfa.harvard.edu> --------- Co-authored-by: Taylor Jacovich <tjacovich@cfa.harvard.edu> * Update update_record to call async task for boost_mesage. Add queue for receiving augment processes. (#215) * Update update_record to call async task for boost_mesage. Add queue for receiving augment processes. * Broke boost call out of update storage to remove circular import. * removed circular import. * Fix args in task_boost_request. * Add awscli back to requirements.txt Added awscli version 1.27.60 to requirements. * Added support for pub_abbrev from import (#217) * Fix extract_classifications_pipeline empty string bug (#219) * Adding has_bib_data label (#216) * Adding has_bib_data label * Adding has_bib_data label * removing comment * removing unnecessary check * Rewrite task_update_record so that calls to task_boost_request is cal… (#220) * Rewrite task_update_record so that calls to task_boost_request is called for individual records when the message is a list type. * Add IGNORED_BOOST_PAYLOAD_TYPES to config. * upgrading scixpipelineutils requirement * Cleared out all classifier code. * update scix ids (#223) --------- Co-authored-by: Fernanda <femalves@users.noreply.github.com> Co-authored-by: mugdhapolimera <35502000+mugdhapolimera@users.noreply.github.com> Co-authored-by: Kelly Lockhart <2926089+kelockhart@users.noreply.github.com> Co-authored-by: Thomas S. Allen <tom.sco@gmail.com> Co-authored-by: Mugdha Polimera <mugdhapolimera@gmail.com>
2 participants
No description provided.