Update blocklist.ts · b8170aaa47 - firecrawl - MKD Repo

Iyeoluwads/firecrawl

Update blocklist.ts

This commit is contained in:

rafaelsideguide

2024-08-19 08:51:48 -03:00

parent 3fe82b4f12

commit b8170aaa47

1 changed files with 2 additions and 1 deletions

									
										apps/api/src/scraper/WebScraper/utils/blocklist.ts
									
		+2
		-1
	
												View File
												
					@@ -15,7 +15,8 @@ const socialMediaBlocklist = [

					  'whatsapp.com',

					  'whatsapp.com',

					  'wechat.com',

					  'wechat.com',

					  'telegram.org',

					  'telegram.org',

					  'researchhub.com'

					  'researchhub.com',

					  'youtube.com'

					];

					];

					const allowedKeywords = [

					const allowedKeywords = [