 Gergő Móricz
|
03b37998fd
|
feat: bulk scrape
|
2024-10-17 19:40:18 +02:00 |
|
 Nicolas
|
d1b838322d
|
Merge pull request #721 from mendableai/feat/concurrency-limit
Concurrency limits
|
2024-10-01 16:15:05 -03:00 |
|
|
Gergő Móricz
|
fe721fffbe
|
fix(crawl-redis): normalize URL before locking
|
2024-10-01 20:59:50 +02:00 |
|
|
Gergő Móricz
|
b696bfc854
|
fix(crawl-status): avoid race conditions where crawl may be deemed failed
|
2024-09-26 21:00:27 +02:00 |
|
|
Nicolas
|
d872bf0c4c
|
Merge branch 'main' into v1-webscraper
|
2024-08-28 12:42:23 -03:00 |
|
|
Nicolas
|
c7bfe4ffe8
|
Nick:
|
2024-08-21 22:20:40 -03:00 |
|
|
Gergő Móricz
|
eb84673b06
|
feat: crawl status websocket WIP
|
2024-08-17 01:04:14 +02:00 |
|
|
Gergő Móricz
|
5896153d19
|
fix: crawl status and redis fixes
|
2024-08-16 22:52:48 +02:00 |
|
|
Gergő Móricz
|
f20328bdbb
|
crawl status and document stuff
|
2024-08-16 22:48:05 +02:00 |
|
|
Gergő Móricz
|
d0a8382a5b
|
fix(queue-worker): crawl finishing race condition
|
2024-08-16 18:48:52 +02:00 |
|
|
Gergő Móricz
|
846610681b
|
fix: fix posthog, add dummy crawl DB items
|
2024-08-15 18:55:18 +02:00 |
|
|
Gergő Móricz
|
b8ec40dd72
|
fix(crawl): submit sitemapped jobs in bulk
|
2024-08-14 20:34:19 +02:00 |
|
|
Gergo Moricz
|
2e5e480cc2
|
fix(crawl): call webhooks
|
2024-08-13 22:10:17 +02:00 |
|
|
Gergo Moricz
|
86e136beca
|
feat: crawl to scrape conversion
|
2024-08-13 20:51:43 +02:00 |
|