Class: SentenceSplitter
SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.
One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.
Constructors
constructor
• new SentenceSplitter(options?): SentenceSplitter
Parameters
| Name | Type |
|---|---|
options? | Object |
options.chunkOverlap? | number |
options.chunkSize? | number |
options.chunkingTokenizerFn? | (text: string) => string[] |
options.paragraphSeparator? | string |
options.splitLongSentences? | boolean |
options.tokenizer? | any |
options.tokenizerDecoder? | any |
Returns
Defined in
packages/core/src/TextSplitter.ts:77
Properties
chunkOverlap
• Private chunkOverlap: number
Defined in
packages/core/src/TextSplitter.ts:70
chunkSize
• Private chunkSize: number
Defined in
packages/core/src/TextSplitter.ts:69
chunkingTokenizerFn
• Private chunkingTokenizerFn: (text: string) => string[]
Type declaration
▸ (text): string[]
Parameters
| Name | Type |
|---|---|
text | string |
Returns
string[]
Defined in
packages/core/src/TextSplitter.ts:74
paragraphSeparator
• Private paragraphSeparator: string
Defined in
packages/core/src/TextSplitter.ts:73
splitLongSentences
• Private splitLongSentences: boolean
Defined in
packages/core/src/TextSplitter.ts:75
tokenizer
• Private tokenizer: any
Defined in
packages/core/src/TextSplitter.ts:71
tokenizerDecoder
• Private tokenizerDecoder: any
Defined in
packages/core/src/TextSplitter.ts:72
Methods
combineTextSplits
▸ combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]
Parameters
| Name | Type |
|---|---|
newSentenceSplits | SplitRep[] |
effectiveChunkSize | number |
Returns
TextSplit[]
Defined in
packages/core/src/TextSplitter.ts:214
getEffectiveChunkSize
▸ getEffectiveChunkSize(extraInfoStr?): number
Parameters
| Name | Type |
|---|---|
extraInfoStr? | string |
Returns
number
Defined in
packages/core/src/TextSplitter.ts:113
getParagraphSplits
▸ getParagraphSplits(text, effectiveChunkSize?): string[]
Parameters
| Name | Type |
|---|---|
text | string |
effectiveChunkSize? | number |
Returns
string[]
Defined in
packages/core/src/TextSplitter.ts:130
getSentenceSplits
▸ getSentenceSplits(text, effectiveChunkSize?): string[]
Parameters
| Name | Type |
|---|---|
text | string |
effectiveChunkSize? | number |
Returns
string[]
Defined in
packages/core/src/TextSplitter.ts:156
processSentenceSplits
▸ processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]
Splits sentences into chunks if necessary.
This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.
Parameters
| Name | Type |
|---|---|
sentenceSplits | string[] |
effectiveChunkSize | number |
Returns
SplitRep[]
Defined in
packages/core/src/TextSplitter.ts:185
splitText
▸ splitText(text, extraInfoStr?): string[]
Parameters
| Name | Type |
|---|---|
text | string |
extraInfoStr? | string |
Returns
string[]
Defined in
packages/core/src/TextSplitter.ts:308
splitTextWithOverlaps
▸ splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]
Parameters
| Name | Type |
|---|---|
text | string |
extraInfoStr? | string |
Returns
TextSplit[]