moving js-sdk to monorepo

2024-04-16 14:02:16 -03:00
parent 0113d66739
commit 3e4064bce2
7 changed files with 879 additions and 0 deletions
@@ -0,0 +1,148 @@
+# Firecrawl JavaScript SDK
+
+The Firecrawl JavaScript SDK is a library that allows you to easily scrape and crawl websites, and output the data in a format ready for use with language models (LLMs). It provides a simple and intuitive interface for interacting with the Firecrawl API.
+
+## Installation
+
+To install the Firecrawl JavaScript SDK, you can use npm:
+
+```bash
+npm install @mendable/firecrawl-js
+```
+
+## Usage
+
+1. Get an API key from [firecrawl.dev](https://firecrawl.dev)
+2. Set the API key as an environment variable named `FIRECRAWL_API_KEY` or pass it as a parameter to the `FirecrawlApp` class.
+
+
+Here's an example of how to use the SDK with error handling:
+
+```js
+  import FirecrawlApp from '@mendable/firecrawl-js';
+
+  async function main() {
+    try {
+      // Initialize the FirecrawlApp with your API key
+      const app = new FirecrawlApp({ apiKey: "YOUR_API_KEY" });
+
+      // Scrape a single URL
+      const url = 'https://mendable.ai';
+      const scrapedData = await app.scrapeUrl(url);
+      console.log(scrapedData);
+      
+      // Crawl a website
+      const crawlUrl = 'https://mendable.ai';
+      const crawlParams = {
+      crawlerOptions: {
+        excludes: ['blog/'],
+        includes: [], // leave empty for all pages 
+        limit: 1000,
+        }
+      };
+
+      const crawlResult = await app.crawlUrl(crawlUrl, crawlParams);
+      console.log(crawlResult);
+
+    } catch (error) {
+      console.error('An error occurred:', error.message);
+    }
+  }
+
+  main();
+```
+
+### Scraping a URL
+
+To scrape a single URL with error handling, use the `scrapeUrl` method. It takes the URL as a parameter and returns the scraped data as a dictionary.
+
+```js
+  async function scrapeExample() {
+    try {
+      const url = 'https://example.com';
+      const scrapedData = await app.scrapeUrl(url);
+      console.log(scrapedData);
+
+    } catch (error) {
+      console.error(
+        'Error occurred while scraping:',
+        error.message
+      );
+    }
+  }
+  
+  scrapeExample();
+```
+
+
+### Crawling a Website
+
+To crawl a website with error handling, use the `crawlUrl` method. It takes the starting URL and optional parameters as arguments. The `params` argument allows you to specify additional options for the crawl job, such as the maximum number of pages to crawl, allowed domains, and the output format.
+
+```js
+async function crawlExample() {
+  try {
+    const crawlUrl = 'https://example.com';
+    const crawlParams = {
+      crawlerOptions: {
+        excludes: ['blog/'],
+        includes: [], // leave empty for all pages
+        limit: 1000,
+      }
+    };
+    const waitUntilDone = true;
+    const timeout = 5;
+    const crawlResult = await app.crawlUrl(
+      crawlUrl,
+      crawlParams,
+      waitUntilDone,
+      timeout
+    );
+
+    console.log(crawlResult);
+
+  } catch (error) {
+    console.error(
+      'Error occurred while crawling:',
+      error.message
+    );
+  }
+}
+
+crawlExample();
+```
+
+
+### Checking Crawl Status
+
+To check the status of a crawl job with error handling, use the `checkCrawlStatus` method. It takes the job ID as a parameter and returns the current status of the crawl job.
+
+```js
+async function checkStatusExample(jobId) {
+  try {
+    const status = await app.checkCrawlStatus(jobId);
+    console.log(status);
+
+  } catch (error) {
+    console.error(
+      'Error occurred while checking crawl status:',
+      error.message
+    );
+  }
+}
+// Example usage, assuming you have a jobId
+checkStatusExample('your_job_id_here');
+```
+
+
+## Error Handling
+
+The SDK handles errors returned by the Firecrawl API and raises appropriate exceptions. If an error occurs during a request, an exception will be raised with a descriptive error message. The examples above demonstrate how to handle these errors using `try/catch` blocks.
+
+## Contributing
+
+Contributions to the Firecrawl JavaScript SDK are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.
+
+## License
+
+The Firecrawl JavaScript SDK is open-source and released under the [MIT License](https://opensource.org/licenses/MIT).