Struct icu::segmenter::WordSegmenter

pub struct WordSegmenter { /* private fields */ }

Supports loading word break data, and creating word break iterators for different string encodings.

🚧 This code is experimental; it may change at any time, in breaking or non-breaking ways, including in SemVer minor releases. It can be enabled with the "experimental" Cargo feature of the icu meta-crate. Use with caution. #2259

Examples

Segment a string:

use icu_segmenter::WordSegmenter;
let segmenter =
    WordSegmenter::try_new_unstable(&icu_testdata::unstable())
        .expect("Data exists");

let breakpoints: Vec<usize> =
    segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);

Segment a Latin1 byte string:

use icu_segmenter::WordSegmenter;
let segmenter =
    WordSegmenter::try_new_unstable(&icu_testdata::unstable())
        .expect("Data exists");

let breakpoints: Vec<usize> =
    segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);

Implementations

impl WordSegmenter

pub fn try_new_unstable<D>(
    provider: &D
) -> Result<WordSegmenter, SegmenterError> where
    D: DataProvider<WordBreakDataV1Marker> + DataProvider<UCharDictionaryBreakDataV1Marker> + DataProvider<LstmDataV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized, 

Construct a WordSegmenter.

pub fn try_new_with_any_provider(
    provider: &impl AnyProvider
) -> Result<WordSegmenter, SegmenterError>

Creates a new instance using an AnyProvider.

For details on the behavior of this function, see: Self::try_new_unstable

📚 Help choosing a constructor

pub fn try_new_with_buffer_provider(
    provider: &impl BufferProvider
) -> Result<WordSegmenter, SegmenterError>

✨ Enabled with the "serde" feature.

Creates a new instance using a BufferProvider.

For details on the behavior of this function, see: Self::try_new_unstable

📚 Help choosing a constructor

pub fn segment_str(
    &'l self,
    input: &'s str
) -> RuleBreakIterator<'l, 's, WordBreakTypeUtf8>

Create a word break iterator for an str (a UTF-8 string).

pub fn segment_utf8(
    &'l self,
    input: &'s [u8]
) -> RuleBreakIterator<'l, 's, WordBreakTypePotentiallyIllFormedUtf8>

Create a word break iterator for a potentially ill-formed UTF8 string

Invalid characters are treated as REPLACEMENT CHARACTER

pub fn segment_latin1(
    &'l self,
    input: &'s [u8]
) -> RuleBreakIterator<'l, 's, RuleBreakTypeLatin1>

Create a word break iterator for a Latin-1 (8-bit) string.

pub fn segment_utf16(
    &'l self,
    input: &'s [u16]
) -> RuleBreakIterator<'l, 's, WordBreakTypeUtf16>

Create a word break iterator for a UTF-16 string.