Skip to content

Instantly share code, notes, and snippets.

@yoke233
yoke233 / testRegex.js
Created August 17, 2024 05:15
Regex for chunking by using all semantic cues
// Updated: Aug. 15, 2024
// Run: node testRegex.js testText.txt
// Used in https://jina.ai/tokenizer
const fs = require('fs');
const util = require('util');
// Define variables for magic numbers
const MAX_HEADING_LENGTH = 7;
const MAX_HEADING_CONTENT_LENGTH = 200;
const MAX_HEADING_UNDERLINE_LENGTH = 200;
// Website you intended to retrieve for users.
const upstream = 'api.openai.com'
// Custom pathname for the upstream website.
const upstream_path = '/'
// Website you intended to retrieve for users using mobile devices.
const upstream_mobile = upstream
// Countries and regions where you wish to suspend your service.