 Gergő Móricz
|
5c62bb1195
|
feat: new snips test framework (FIR-414) (#1033)
* feat: new snips test framework
* Update mock.ts
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
|
2025-01-13 20:50:47 +01:00 |
|
 Nicolas
|
f4d10c5031
|
Nick: formatting fixes
|
2025-01-10 18:35:10 -03:00 |
|
|
Gergő Móricz
|
d1f3b96388
|
feat: add scrapeId in document.metadata
|
2025-01-09 20:52:12 +01:00 |
|
|
Gergő Móricz
|
29c1f126ab
|
feat(scrape-status): adapt
|
2025-01-09 19:14:00 +01:00 |
|
|
Nicolas
|
14f696805c
|
Update auth.ts
|
2025-01-08 17:04:57 -03:00 |
|
|
Nicolas
|
f82a742cd1
|
Merge pull request #1044 from mendableai/nsc/extract-queue
(feat/extract) Move extract to a queue system
|
2025-01-07 18:10:46 -03:00 |
|
|
Nicolas
|
b98e289f03
|
Nick:
|
2025-01-07 17:49:21 -03:00 |
|
|
Nicolas
|
9ec08d7020
|
Nick: fixed the sdks
|
2025-01-07 17:20:49 -03:00 |
|
|
Nicolas
|
dd14744850
|
Update types.ts
|
2025-01-07 16:55:55 -03:00 |
|
|
Nicolas
|
11af214db1
|
Nick: update extract in case there is an error
|
2025-01-07 16:21:51 -03:00 |
|
|
Nicolas
|
eb254547e5
|
Nick:
|
2025-01-07 16:16:01 -03:00 |
|
|
Gergő Móricz
|
ccfada98ca
|
various queue fixes
|
2025-01-07 19:15:23 +01:00 |
|
|
Nicolas
|
86e34d7c6c
|
Nick: wip
|
2025-01-07 12:13:12 -03:00 |
|
|
Móricz Gergő
|
b96b97ed72
|
fix(crawl): don't push rawhtml to db unless requested
|
2025-01-07 10:09:15 +01:00 |
|
|
Nicolas
|
bb27594443
|
Merge branch 'main' into nsc/extract-queue
|
2025-01-06 13:01:15 -03:00 |
|
|
Gergő Móricz
|
461842fe8c
|
fix(v1/crawl-status): handle job's returnvalue being explicitly null (db race)
|
2025-01-04 17:24:33 +01:00 |
|
|
Gergő Móricz
|
b92a4eb79b
|
fix(queue-worker): only do redirect handling logic on crawls, not batch scrape
|
2025-01-04 16:59:35 +01:00 |
|
|
Nicolas
|
27457ed5db
|
Nick: init
|
2025-01-03 20:44:27 -03:00 |
|
|
Nicolas
|
ad49503f8a
|
Update search.ts
|
2025-01-02 21:15:47 -03:00 |
|
|
Nicolas
|
cbe0716439
|
Update search.ts
|
2025-01-02 21:13:24 -03:00 |
|
|
Nicolas
|
e37ab8431a
|
Update search.ts
|
2025-01-02 21:07:14 -03:00 |
|
|
Nicolas
|
8b64e915b3
|
Update search.ts
|
2025-01-02 21:02:55 -03:00 |
|
|
Nicolas
|
7ce780ac81
|
Update search.ts
|
2025-01-02 20:40:38 -03:00 |
|
|
Nicolas
|
21bf89b6cc
|
Update search.ts
|
2025-01-02 19:57:51 -03:00 |
|
|
Nicolas
|
22ae1730bd
|
Update search.ts
|
2025-01-02 19:57:41 -03:00 |
|
|
Nicolas
|
a0dbf20c40
|
Update types.ts
|
2025-01-02 19:55:28 -03:00 |
|
|
Nicolas
|
35d7202894
|
Update search.ts
|
2025-01-02 19:33:21 -03:00 |
|
|
Nicolas
|
d2742bec4d
|
Nick: v1 search
|
2025-01-02 19:31:03 -03:00 |
|
|
Nicolas
|
0847a6038e
|
Merge pull request #1014 from mendableai/nsc/extract-url-trace
/extract URL trace
|
2024-12-30 19:00:58 -03:00 |
|
|
Gergő Móricz
|
0421f81020
|
Sitemap fixes (#1010)
* sitemap fixes iter 1
* feat(sitemap): dedupe improvements
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
|
2024-12-27 19:59:26 +01:00 |
|
|
Nicolas
|
4332f18a8f
|
Nick: making it optional for the user
|
2024-12-26 12:43:58 -03:00 |
|
|
Nicolas
|
233f347f5e
|
Nick: refactor
|
2024-12-26 12:41:37 -03:00 |
|
|
Nicolas
|
f467a3ae6c
|
Nick: init
|
2024-12-26 12:21:46 -03:00 |
|
|
Nicolas
|
d1f3e26f9e
|
Nick: blocklist string
|
2024-12-20 18:09:49 -03:00 |
|
|
Nicolas
|
6222152249
|
Nick: credit usage endpoint
|
2024-12-20 15:44:17 -03:00 |
|
|
Nicolas
|
05605112bb
|
Update extract.ts
|
2024-12-18 23:34:07 -03:00 |
|
|
Nicolas
|
2d37dca9dc
|
Nick: introduced system prompt to /extract
|
2024-12-18 22:10:41 -03:00 |
|
|
Nicolas
|
a759a7ab7a
|
Nick: small improvements
|
2024-12-18 21:45:06 -03:00 |
|
|
Móricz Gergő
|
780442d73b
|
feat: improve billing logging
|
2024-12-17 22:02:31 +01:00 |
|
|
Nicolas
|
ac187452c3
|
Nick: better filtering for urls that should be scraped
|
2024-12-17 17:34:55 -03:00 |
|
|
Nicolas
|
3b6edef9fa
|
chore: formatting
|
2024-12-17 16:58:57 -03:00 |
|
|
Nicolas
|
b9f621bed5
|
Nick: extract fixes
|
2024-12-17 16:58:35 -03:00 |
|
|
Nicolas
|
79e335636a
|
Nick: fixed extract issues
|
2024-12-17 16:40:45 -03:00 |
|
|
Nicolas
|
6d77879d68
|
Update extract.ts
|
2024-12-17 15:22:25 -03:00 |
|
|
Nicolas
|
e26a0a65a7
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-12-17 15:19:53 -03:00 |
|
|
Nicolas
|
0f8b8a717d
|
Update map.ts
|
2024-12-17 15:19:52 -03:00 |
|
|
Gergő Móricz
|
0013bdfcb4
|
feat(v1/scrape): add more context to timeout logs
|
2024-12-16 22:42:51 +01:00 |
|
|
Gergő Móricz
|
2de659d810
|
fix(queue-jobs): fix concurrency limit
|
2024-12-15 23:54:52 +01:00 |
|
|
Gergő Móricz
|
842b522b44
|
feat: add scrapeOptions.fastMode
|
2024-12-15 14:28:47 +01:00 |
|
|
Nicolas
|
588f747ee8
|
chore: formatting
|
2024-12-15 02:54:49 -03:00 |
|