Slice a string containing Unicode chars

Slice a string containing Unicode chars



I have a piece of text with characters of different bytelength.


let text = "Hello привет";



I need to take a slice of the string given start (included) and end (excluded) character indices. I tried this


let slice = &text[start..end];



and got the following error


thread 'main' panicked at 'byte index 7 is not a char boundary; it is inside 'п' (bytes 6..8) of `Hello привет`'



I suppose it happens since Cyrillic letters are multi-byte and the [..] notation takes chars using byte indices. What can I use if I want to slice using character indices, like I do in Python:


[..]



slice = text[start:end] ?


slice = text[start:end]



I know I can use the chars() iterator and manually walk through the desired substring, but is there a more concise way?


chars()





I think chars() is the way to go here: text.chars().take(end).skip(start)
– Tim Diekmann
Aug 23 at 10:10


chars()


text.chars().take(end).skip(start)





@TimDiekmann how do I convert the Take<Chars> to &str then if the API needs it?
– Sasha Tsukanov
Aug 23 at 10:17


Take<Chars>


&str





You should call collect(). See this question stackoverflow.com/questions/37157926/…
– ozkriff
Aug 23 at 10:18



collect()





@ozkriff collect() will result in String, not in &str. This is why I didn't marked this as duplicate to your linked question.
– Tim Diekmann
Aug 23 at 10:31


collect()


String


&str




2 Answers
2



I know I can use the chars() iterator and manually walk through the desired substring, but is there a more concise way?


chars()



If you know the exact byte indices, you can slice a string:


let text = "Hello привет";
println!("", &text[2..10]);



This prints "llo пр". So the problem is to find out the exact byte position. You can do that fairly easily with the char_indices() iterator (alternatively you could use chars() with char::len_utf8()):


char_indices()


chars()


char::len_utf8()


let text = "Hello привет";
let end = text.char_indices().map(|(i, _)| i).nth(8).unwrap();
println!("", &text[2..idx]);



As another alternative, you can first collect the string into Vec<char>. Then, indexing is simple, but to print it as a string, you have to collect it again or write your own function to do it.


Vec<char>


let text = "Hello привет";
let text_vec = text.chars().collect::<Vec<_>>();
println!("", text_vec[2..8].iter().cloned().collect::<String>());



As you can see, neither of these solutions is all that great. This is intentional, for two reasons:



As str is a simply UTF8 buffer, indexing by unicode codepoints is an O(n) operation. Usually, people expect the operator to be a O(1) operation. Rust makes this runtime complexity explicit and doesn't try to hide it. In both solutions above you can clearly see that it's not O(1).


str




But the more important reason:



What Python does (and what you think you want) is not all that useful. It all comes down to the complexity of language and thus the complexity of unicode. Python slices Unicode codepoints. This is what a Rust char represents. It's 32 bit big (a few fewer bits would suffice, but we round up to a power of 2).


char



But what you actually want to do is slice user perceived characters. But this is an explicitly loosely defined term. Different cultures and languages regard different things as "one character". The closest approximation is a "grapheme cluster". Such a cluster can consist of one or more unicode codepoints. Consider this Python 3 code:


>>> s = "Jürgen"
>>> s[0:2]
'Ju'



Surprising, right? This is because the string above is:


0x004A


0x0075


0x0308



This is an example of a combining character that is rendered as part of the previous character. Python slicing does the "wrong" thing here.



Another example:


>>> s = "fire"
>>> s[0:2]
'fir'



Also not what you'd expect. This time, fi is actually the ligature , which is one codepoint.


fi




There are far more examples where Unicode behaves in a surprising way. See the links at the bottom for more information and examples.



So if you want to work with international strings that should be able to work everywhere, don't do codepoint slicing! If you really need to semantically view the string as a series of characters, use grapheme clusters. To do that, the crate unicode-segmentation is very useful.


unicode-segmentation



Further resources on this topic:





To make let end = text.char_indices().map(|(i, _)| i).nth(8).unwrap(); work when we want to slice till the last codepoint in the string (say, with index 11) by using, say, 12 as excluded bound we need more work. One can add something like let end = if end_codepoint_idx == text.chars().count() text.len() else (i, _);
– Sasha Tsukanov
Aug 23 at 13:58


let end = text.char_indices().map(|(i, _)| i).nth(8).unwrap();


let end = if end_codepoint_idx == text.chars().count() text.len() else (i, _);



An UTF-8 encoded string may contain characters, which consists of multiple bytes. In your case, п starts at index 6 (inclusive) and ends at position 8 (exclusive) so indexing 7 is not the start of the character. This is why your error occurred.


п



You may use str::char_indices for solving this (remember, that getting to a position in UTF-8 is O(n)):


str::char_indices


O(n)


fn get_utf8_slice(string: &str, start: usize, end: usize) -> Option<&str> (end_pos, _))



playground



You may use str::chars() if you are fine with getting a String:


str::chars()


String


let string: String = text.chars().take(end).skip(start).collect();






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Popular posts from this blog

𛂒𛀶,𛀽𛀑𛂀𛃧𛂓𛀙𛃆𛃑𛃷𛂟𛁡𛀢𛀟𛁤𛂽𛁕𛁪𛂟𛂯,𛁞𛂧𛀴𛁄𛁠𛁼𛂿𛀤 𛂘,𛁺𛂾𛃭𛃭𛃵𛀺,𛂣𛃍𛂖𛃶 𛀸𛃀𛂖𛁶𛁏𛁚 𛂢𛂞 𛁰𛂆𛀔,𛁸𛀽𛁓𛃋𛂇𛃧𛀧𛃣𛂐𛃇,𛂂𛃻𛃲𛁬𛃞𛀧𛃃𛀅 𛂭𛁠𛁡𛃇𛀷𛃓𛁥,𛁙𛁘𛁞𛃸𛁸𛃣𛁜,𛂛,𛃿,𛁯𛂘𛂌𛃛𛁱𛃌𛂈𛂇 𛁊𛃲,𛀕𛃴𛀜 𛀶𛂆𛀶𛃟𛂉𛀣,𛂐𛁞𛁾 𛁷𛂑𛁳𛂯𛀬𛃅,𛃶𛁼

Crossroads (UK TV series)

ữḛḳṊẴ ẋ,Ẩṙ,ỹḛẪẠứụỿṞṦ,Ṉẍừ,ứ Ị,Ḵ,ṏ ṇỪḎḰṰọửḊ ṾḨḮữẑỶṑỗḮṣṉẃ Ữẩụ,ṓ,ḹẕḪḫỞṿḭ ỒṱṨẁṋṜ ḅẈ ṉ ứṀḱṑỒḵ,ḏ,ḊḖỹẊ Ẻḷổ,ṥ ẔḲẪụḣể Ṱ ḭỏựẶ Ồ Ṩ,ẂḿṡḾồ ỗṗṡịṞẤḵṽẃ ṸḒẄẘ,ủẞẵṦṟầṓế