Details
-
Task
-
Resolution: Fixed
-
P0: Blocker
-
None
-
None
-
None
Description
The coin_scraper.py script currently asserts because Coin returns newly unseen tasks, in-between other previously seen tasks ( COIN-965 ). Tasks are always ordered in started_on date in the response, which means that some tasks are added into the list after they started.
The script fetches tasks periodically. Each time, it fetches up to the most recently seen task. Since new tasks can appear after that one, there is no clear point that the script knows it has to stop fetching.
Suggested Workaround
The suggested workaround is to fetch another 1000 tasks after the last fetched task, and for each one of the extra tasks, check if we already have it or not.
Attachments
Issue Links
- resulted from
-
COIN-965 GET /tasks responses are not stable
- Need More Info