 rafaelsideguide
|
4d6e25619b
|
minor spacing and comment stuff
|
2024-07-01 16:05:34 -03:00 |
|
 Jeff Pereira
|
a5fb45988c
|
new feature allowExternalContentLinks
|
2024-06-28 17:23:40 -07:00 |
|
 Nicolas
|
9bf74bc774
|
Update single_url.ts
|
2024-06-28 15:51:18 -03:00 |
|
|
Nicolas
|
7e17498bcf
|
Update single_url.ts
|
2024-06-28 15:45:16 -03:00 |
|
|
Nicolas
|
042f81ddf2
|
Update removeUnwantedElements.test.ts
|
2024-06-26 21:20:11 -03:00 |
|
|
Nicolas
|
388ce3cbce
|
Nick: small changes
|
2024-06-26 21:15:42 -03:00 |
|
|
Nicolas
|
1d4907acc9
|
Nick:
|
2024-06-26 21:02:58 -03:00 |
|
|
Nicolas
|
3b92fb8433
|
Merge pull request #322 from mendableai/tests/metadata
[Test] Added E2E tests for checking metadata values
|
2024-06-26 12:09:18 -03:00 |
|
|
rafaelsideguide
|
67d7650cf3
|
Added to e2e_noAuth
|
2024-06-26 12:07:55 -03:00 |
|
|
rafaelsideguide
|
05eaa3c68d
|
Update index.test.ts
|
2024-06-26 09:32:02 -03:00 |
|
|
rafaelsideguide
|
4381109dd8
|
added default values and fixed pdf bug
|
2024-06-26 09:00:54 -03:00 |
|
|
Nicolas
|
45f2765601
|
Merge pull request #316 from snippet/types-webscraper
add some types
|
2024-06-25 22:03:21 -03:00 |
|
|
Nicolas
|
768a131b5c
|
Merge pull request #318 from mendableai/bug/fix-custom-scrape-pdf-google-drive
[Bug] Fixed the regex test for google drive pdf files
|
2024-06-25 18:27:11 -03:00 |
|
|
rafaelsideguide
|
5f69fc7677
|
Fixed the regex test
|
2024-06-25 18:24:01 -03:00 |
|
|
rafaelsideguide
|
d02829d335
|
fixed clean jobs
|
2024-06-25 17:49:29 -03:00 |
|
|
Jeff Pereira
|
199cbe8bcb
|
add some types
|
2024-06-25 12:20:25 -07:00 |
|
|
Nicolas
|
749b0c05dc
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-06-25 15:21:15 -03:00 |
|
|
Nicolas
|
e7be17db92
|
Nick: metadata fixes and lock duration for bull decreased to 2 hrs
|
2024-06-25 15:21:14 -03:00 |
|
|
Nicolas
|
f84fb4b331
|
Merge pull request #313 from snippet/google-search-term-fix
fix multi-word search term issue: /search (w/o Serp)
|
2024-06-24 19:24:58 -03:00 |
|
|
Jeff Pereira
|
6ddf3a58a1
|
fix multi-word search term issue: /search (w/o Serp)
|
2024-06-24 14:21:52 -07:00 |
|
|
Nicolas
|
90b7fff366
|
Update crawler.ts
|
2024-06-24 16:52:01 -03:00 |
|
|
Nicolas
|
08c1fa799b
|
Update queue-worker.ts
|
2024-06-24 16:51:32 -03:00 |
|
|
rafaelsideguide
|
3ebdf93342
|
removed console.logs
|
2024-06-24 16:43:12 -03:00 |
|
|
Nicolas
|
56d42d9c9b
|
Nick:
|
2024-06-24 16:33:07 -03:00 |
|
|
rafaelsideguide
|
21d29de819
|
testing crawl with new.abb.com case
many unnecessary console.logs for tracing the code execution
|
2024-06-24 16:25:07 -03:00 |
|
|
Nicolas
|
3c7b7e7242
|
NIck: fixes fallback
|
2024-06-23 18:59:08 -03:00 |
|
 Caleb Peffer
|
e59ba758f5
|
Caleb: changed posthog logging so that It associates jobs with a group. No
|
2024-06-18 17:42:21 -07:00 |
|
|
Caleb Peffer
|
5a91d8425f
|
Caleb: solve for typechecking on idempotencyKey on my machine
|
2024-06-18 17:07:38 -07:00 |
|
|
rafaelsideguide
|
9c539e9113
|
Fixed includeHTML to use cleanedHtml as response
|
2024-06-18 16:26:54 -03:00 |
|
|
Rafael Miller
|
f5a9acc4c6
|
Merge branch 'main' into feat/removeTags-regex
|
2024-06-18 14:39:59 -03:00 |
|
|
rafaelsideguide
|
9f7afd1e88
|
fix for some complex cases
|
2024-06-18 14:36:51 -03:00 |
|
|
Nicolas
|
d0c05accf6
|
Nick:
|
2024-06-18 13:21:50 -04:00 |
|
|
Nicolas
|
818751a256
|
Merge pull request #294 from mendableai/tests/e2e-to-unit
[Test] Transcribed from e2e to unit tests for many cases
|
2024-06-18 13:09:22 -04:00 |
|
|
rafaelsideguide
|
727e5de8c5
|
Update index.test.ts
|
2024-06-18 11:54:10 -03:00 |
|
|
rafaelsideguide
|
20f14bcf7f
|
Added some types
|
2024-06-18 10:55:07 -03:00 |
|
|
rafaelsideguide
|
c2fc69af1c
|
removed some e2e tests that are making the ci get stuck
|
2024-06-18 09:57:05 -03:00 |
|
|
rafaelsideguide
|
6c726a02eb
|
Moved to utils/removeUnwantedElements, added unit tests
|
2024-06-18 09:46:42 -03:00 |
|
 AndyMik90
|
8b3c3aae91
|
Added support for RegEx in removeTags
|
2024-06-18 07:31:46 +02:00 |
|
|
rafaelsideguide
|
b2bd562bb2
|
transcribed from e2e to unit tests for many cases
|
2024-06-17 17:09:44 -03:00 |
|
|
Nicolas
|
ab038051e9
|
Merge branch 'main' into nsc/rate-limiter-tests
|
2024-06-17 15:06:12 -04:00 |
|
 Eric Ciarla
|
519ab1aecb
|
Update unit tests
|
2024-06-15 17:14:09 -04:00 |
|
|
Eric Ciarla
|
f0d4146b42
|
Merge branch 'feat/maxDepthRelative' of https://github.com/mendableai/firecrawl into feat/maxDepthRelative
|
2024-06-15 16:52:00 -04:00 |
|
|
Eric Ciarla
|
ff7b52cab1
|
Delete one more e2e test
|
2024-06-15 16:51:50 -04:00 |
|
 Eric Ciarla
|
b1eb608295
|
Merge branch 'main' into feat/maxDepthRelative
|
2024-06-15 16:50:27 -04:00 |
|
|
Eric Ciarla
|
34e37c5671
|
Add unit tests to replace e2e
|
2024-06-15 16:43:37 -04:00 |
|
|
Eric Ciarla
|
2b40729cc2
|
Update index.test.ts
|
2024-06-15 08:56:32 -04:00 |
|
|
Eric Ciarla
|
f22759b2e7
|
Update index.test.ts
|
2024-06-14 19:42:11 -04:00 |
|
|
Eric Ciarla
|
a6b7197737
|
Fix for maxDepth
|
2024-06-14 19:40:37 -04:00 |
|
|
Nicolas
|
4ec863718b
|
Merge pull request #283 from mendableai/nsc/crawler-fixes
Fixes crawler getting confused with base paths that contain www.
|
2024-06-14 13:50:32 -07:00 |
|
|
Nicolas
|
43767360d8
|
Merge branch 'main' into nsc/rate-limiter-tests
|
2024-06-14 13:50:21 -07:00 |
|