Commit Graph

46 Commits

Author SHA1 Message Date
ecebb100b7 Route carousel media by URL type to avoid double-downloading videos
Carousels return a mixed image+video URL list. The uploader keyed on the
post's media_type, so every carousel URL went through the image optimizer;
videos failed PIL ('cannot identify image file') and were then re-downloaded
for the raw fallback. That double download per video pushed the synchronous
scrape past the backend's request timeout, so the backend imported 0 while
the engine still saved posts to its own DB (causing a dedup desync on retry).
Detect video URLs up front and send them straight to the video uploader.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 11:01:29 +00:00
62a03c61fe fix(scraper): always fetch newest posts first
Stop resuming from the persisted (older) BoxAPI cursor. Resuming made every
sync walk further back in time and never re-fetch posts published since the
last run, so a seller's newest posts were never imported. Always start from
max_id=None (newest) and page older within a single run, de-duping against
the DB: brand-new posts first, then older un-imported ones.

Also emit timezone-aware UTC created_at so ordering is correct regardless of
the container's local timezone (and Django no longer receives naive datetimes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 07:28:23 +00:00
bc965a0cbd khastam 2025-09-25 03:58:33 +03:30
70835c2b64 shayad hal shode bashe 2025-09-25 03:45:40 +03:30
cae8633ec1 fek konam akharishe 2025-09-25 03:29:41 +03:30
f872304db2 jane namoset 2025-09-25 03:17:33 +03:30
7ae458f626 f 2025-09-25 03:08:47 +03:30
760568e571 fuck you 2025-09-25 03:03:19 +03:30
812d9964b4 dd 2025-09-25 02:56:48 +03:30
bbc9bc670d ok 2025-09-25 02:52:44 +03:30
90c28032cc ishalla fixe 2025-09-25 02:44:14 +03:30
2e55f2d99e enought 2025-09-25 02:37:17 +03:30
79b87a5551 update 2025-09-25 02:34:01 +03:30
c80a10680b update 2025-09-25 02:26:58 +03:30
ba0c636ffc update 2025-09-25 02:19:42 +03:30
898705ed51 update 2025-09-25 02:15:20 +03:30
8020a8aff3 done 2025-09-23 12:42:20 +03:30
e72dfa7c1a d 2025-09-23 12:31:09 +03:30
cfa1a2fadb update 2025-09-23 12:23:56 +03:30
2de6160d58 update somthing 2025-09-22 15:00:58 +03:30
20954291c0 add ffmpeg 2025-09-16 20:14:01 +03:30
d46d322210 update the sync 2025-08-11 23:38:53 +03:30
6b172b4fc0 migrate the data 2025-08-11 23:35:16 +03:30
9349633feb update the scrapper 2025-08-11 23:12:14 +03:30
ce86d474a3 update the scrapper.py 2025-08-11 22:58:46 +03:30
49edb4a8cf update schema 2025-08-11 22:54:43 +03:30
bec32b4df2 update 2025-08-11 22:33:55 +03:30
2026e0fbfb update the docker-compose 2025-08-11 22:24:37 +03:30
d8c349f4fb make it faster 2025-08-11 22:19:23 +03:30
e31df1e3db update 2025-08-11 22:07:49 +03:30
5504b1160d update it 2025-08-11 21:54:56 +03:30
d524171b49 up[date the client 2025-08-11 21:45:31 +03:30
f3b5532abb change the system to boxapi client 2025-08-11 21:42:00 +03:30
6ce50b72ad dodododdo 2025-08-07 03:10:33 +03:30
65ccffd6e2 sssss 2025-08-07 03:02:13 +03:30
48b28e8ff9 sssss 2025-08-07 02:57:10 +03:30
4583e08ca5 scrapper hai hai 2025-08-07 02:46:57 +03:30
02f6ed5b9b scrapper update 2025-08-07 02:41:10 +03:30
8f60e9bcdf Main.py 2025-08-07 01:50:31 +03:30
3b3e9823ae client.py 2025-08-07 01:36:42 +03:30
62743cdb62 MAIN.PY 2025-08-07 01:17:32 +03:30
dd9e639e6d requrements 2025-08-07 01:15:06 +03:30
3fa78e4b3a fix requirements 2025-08-07 01:12:06 +03:30
3761de8de0 FastHOSSEIN 2025-08-07 00:53:03 +03:30
817d3b9c39 create DB 2025-08-07 00:27:44 +03:30
991073ce1b Fist CONDFIGURATIONS 2025-08-07 00:16:18 +03:30