@lg-tools/crawler

Version:

Crawler for MongoDB documentation and other sites

9 files

Filename	Content Type	Size
../
checkRobots.ts	text/typescript	2.64 kB
connectToMongoDB.ts	text/typescript	570 B
createChecksum.ts	text/typescript	262 B
createClusterNameFromURL.ts	text/typescript	567 B
loadPageContents.ts	text/typescript	1.3 kB
mapLangChainDocument.ts	text/typescript	650 B
newURL.ts	text/typescript	213 B
processSingleUrl.ts	text/typescript	3.99 kB
recursiveCrawlFromBase.ts	text/typescript	4.27 kB