Struct icu::segmenter::SentenceSegmenter

pub struct SentenceSegmenter { /* private fields */ }

Supports loading sentence break data, and creating sentence break iterators for different string encodings.

🚧 This code is experimental; it may change at any time, in breaking or non-breaking ways, including in SemVer minor releases. It can be enabled with the "experimental" Cargo feature of the icu meta-crate. Use with caution. #2259

Examples

Segment a string:

use icu_segmenter::SentenceSegmenter;
let segmenter =
    SentenceSegmenter::try_new_unstable(&icu_testdata::unstable())
        .expect("Data exists");

let breakpoints: Vec<usize> =
    segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);

Segment a Latin1 byte string:

use icu_segmenter::SentenceSegmenter;
let segmenter =
    SentenceSegmenter::try_new_unstable(&icu_testdata::unstable())
        .expect("Data exists");

let breakpoints: Vec<usize> =
    segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 11]);

Implementations

impl SentenceSegmenter

pub fn try_new_unstable<D>(
    provider: &D
) -> Result<SentenceSegmenter, SegmenterError> where
    D: DataProvider<SentenceBreakDataV1Marker> + ?Sized, 

Construct a SentenceSegmenter.

pub fn try_new_with_any_provider(
    provider: &impl AnyProvider
) -> Result<SentenceSegmenter, SegmenterError>

Creates a new instance using an AnyProvider.

For details on the behavior of this function, see: Self::try_new_unstable

📚 Help choosing a constructor

pub fn try_new_with_buffer_provider(
    provider: &impl BufferProvider
) -> Result<SentenceSegmenter, SegmenterError>

✨ Enabled with the "serde" feature.

Creates a new instance using a BufferProvider.

For details on the behavior of this function, see: Self::try_new_unstable

📚 Help choosing a constructor

pub fn segment_str(
    &'l self,
    input: &'s str
) -> RuleBreakIterator<'l, 's, RuleBreakTypeUtf8>

Create a sentence break iterator for an str (a UTF-8 string).

pub fn segment_utf8(
    &'l self,
    input: &'s [u8]
) -> RuleBreakIterator<'l, 's, RuleBreakTypePotentiallyIllFormedUtf8>

Create a sentence break iterator for a potentially ill-formed UTF8 string

Invalid characters are treated as REPLACEMENT CHARACTER

pub fn segment_latin1(
    &'l self,
    input: &'s [u8]
) -> RuleBreakIterator<'l, 's, RuleBreakTypeLatin1>

Create a sentence break iterator for a Latin-1 (8-bit) string.

pub fn segment_utf16(
    &'l self,
    input: &'s [u16]
) -> RuleBreakIterator<'l, 's, RuleBreakTypeUtf16>

Create a sentence break iterator for a UTF-16 string.