GeezerVison

UTF-8 Handling for Modern C++  ♦  FastUtf8::Uniseries Guide and Reference

By Kirk J Krauss

What’s here

• A guide to the FastUtf8::Uniseries C++ class outlined in the FastUtf8 overview and defined in fastutf8.cpp;
• A description of each Uniseries operator and method; and
• Example code.

---

Working with FastUtf8

The FastUtf8 namespace provides functionality that allows you to efficiently handle UTF-8 content, adaptively relying on the standard C library for optimized handling of ASCII text.  Methods for handling content are provided via the FastUtf8::Uniseries class.  In addition to covering the fundamentals of working with internationalized content, the class includes methods for searching semistructured content more efficiently than prior ASCII-centric techniques could search through ASCII text, and with simpler coding too.  Usage of each FastUtf8::Uniseries method is described here.

---

The FastUtf8::Uniseries class

An object of the FastUtf8::Uniseries classstores the following data:

• A content buffer, and
• An item of metadata.

The metadata item comprises a small set of flags and a length field.  It fits into a single address on a 64-bit system, for speedy access.  Though that optimization and others underlying the Uniseries class involve a degree of complexity, the class’s methods allow for easily comprehensible management of UTF-8 content and enable these operations:

• Creation, validation, duplication, and deletion of UTF-8 content;
• Slicing (substring extraction) and concatenation of UTF-8 content;
• Optimized slicing and concatenation of ASCII content;
• Separation of UTF-8 content based on UTF-8 delimiters;
• Optimized separation of ASCII content based on ASCII delimiters;
• Content-preserving tokenset search;
• Whole content comparison: case-sensitive and case-insensitive;
• Partial content comparison (basic full-text search): case-sensitive and case-insensitive;
• Wildcard-based content comparison: case-sensitive and case-insensitive;
• Targeted wildcard search: case-sensitive and case-insensitive;
• Slice-within-slice techniques, including
  • Slice search for finding a slice within a larger slice: case-sensitive and case-insensitive,
  • Wildcard search for pattern matching to find a slice within a slice: case-sensitive and case-insensitive,
  • Vectorized wildcard search for finding matches on multiple wildcards: case-sensitive and case-insensitive, and
  • Wild select, for querying hierarchical semistructured content: case-sensitive and case-insensitive;
• Conversion of 8-bit ASCII text to equivalent UTF-8 content;
• Content length and size determination, in code points and in bytes; and
• Trim (removal of outboard white space), index, and case fold operations.

The Uniseries class provides an iterator that can be obtained via the class’s begin() or end() method.  An iterator obtained via end() refers to the position of a terminating null at the end of the content buffer.  The iterator concepts indicate the capability to access content by index, i.e., using operator[], which is implemented as a method of the Uniseries class.  But indexing into UTF-8 content, unless it comprises entirely 7-bit ASCII text, entails quite a performance drag.

To find a code point using the increment and decrement operators, such as operator++, the necessary UTF-8 traversal involves calls to the C-style functions CodePointAdvanceUtf8() and CodePointBacktrackUtf8(), found just in fastutf8.cpp.  Individual code point comparison via operator== involves a call to CodePointCompareUtf8(), also just in fastutf8.cpp.  The other FastUtf8 functionality is performed by C-style functions either declared in fastutf8.h (and also defined in fastutf8.cpp) or available in the standard C library.

---

Flags

The metadata for the Uniseries class includes these flags:

• IS_7BIT_CHAR_STRING;
• IS_CASE_INSENSITIVE;
• IS_LENGTH_LIMITED; and
• IS_DEALLOCATED_EXTERNALLY.

As these names suggest, the flags indicate whether the content is to be treated as 7-bit ASCII, as case-insensitive, and/or as length-limited.  There are length-limited varieties of every relevant C-style function that gets invoked by the Uniseries methods; there also are equivalent – but faster – functions that operate over content until reaching any terminating null.  Which to choose?  See the “Null Terminators Significantly Outperform Length Limits” sidebar in the FastUtf8 overview.

A Uniseries object is constructed with default flag settings that determine its behavior, as follows.  Every data-intensive Uniseries method is designed to handle multi-byte UTF-8 content by invoking a legacy-C-style function also declared in fastutf8.h and defined in fastutf8.cpp.  Many of the Uniseries methods invoke standard C library functions when they’re handling 7-bit ASCII text, or they invoke similar char * functionality, for best ASCII performance.  In most cases, one of several C-style functions is selected, based on settings of the Uniseries object’s flags.  Unless default flag settings are changed, most Uniseries constructors set the flags for use with case-sensitive functionality that expects a terminating null, rather than set up a length check for each access to the object throughout its lifetime.  Regarding that default behavior, the copy constructor is an exception; it copies the flags as part of the Uniseries object being copied.

The IS_DEALLOCATED_EXTERNALLY flag tells the Uniseries destructor to avoid deallocating content that is part of a larger buffer.  This becomes useful if a relatively large buffer is separated into smaller null-terminated portions via the Uniseries::pSeparate() method, which works approximately like the C standard library’s strtok() function, except that it provides for tokenset search, plus it’s thread-safe.

The metadata also includes the length of the Uniseries object’s content buffer.  There are 60 bits available for the content length, so it can represent content that’s up to 152 quadrillion code points in length.  The length is specified as a count of code points, not bytes.

---

Public functionality

The Uniseries content and metadata is declared as private in fastutf8.h.  All other Uniseries operators and methods are public, and all reside in the FastUtf8 namespace.  In the example code provided here, they are presented as if the directive...

using namespace FastUtf8;

...is in place, for brevity.  All Uniseries content is intended to be compatible with 7-bit ASCII or with UTF-8.  The Uniseries arrangements are not compatible with other natural language encodings.

---

Uniseries construction and destruction

Parameterized Constructors

This constructor validates the content in the inbound buffer and makes a deep copy by invoking C-style functions implemented in fastutf8.cpp.  If the content includes any invalid code point(s), the constructor treats the entire buffer as 8-bit ASCII and converts it, as such, into valid UTF-8 content.

Signature

Parameters

Discussion

---

Standard constructor

The standard constructor creates an empty Uniseries object with a content buffer capacity specified in bytes.  The metadata has no flags set upon construction.

Signature

Parameter

---

Copy constructor

This constructor creates a Uniseries object from the content buffer and metadata of an existing one, with no validation, performing a deep copy like the parameterized constructors.

Signature

Parameter

---

Range-based slice constructors

Either of these constructors creates a Uniseries object from a buffer, or from a portion of a buffer, designated by the pFirst and pLast pointers.

Signature

Parameters

Discussion

---

Destructor

The destructor deallocates the content buffer.

Signature

---

Uniseries assignment operators

Assignment of raw buffer content

These assignment operators replace the content and metadata associated with an existing Uniseries object, first validating the content in the inbound buffer and then making a deep copy by calling the CopyUtf8() function.  If the content includes any invalid code point(s), the operator treats the entire buffer as 8-bit ASCII and converts it, via Convert8BitAsciiToUtf8(), to valid UTF-8 content.

Signature

Parameter

---

Assignment of Uniseries content

These assignment operators replace the content and metadata associated with the current Uniseries object with the content buffer and metadata of another existing one, with no validation, performing a deep copy like the copy constructor.

Signature

Parameter

Example: Tokenset search demo

The UniseriespFindTokenDemo() function is part of the tokenset search demo (TokensetSearchDemo.cpp).  Output from the demo program is shown in Fig. 2.

// C++ demo for UTF-8-ready routines.
//
// Certain functions in this file comprise machine-generated code and are 
// described as such in the relevant comments.  All other code in this file 
// is copyright 2026 Kirk J Krauss and is a Derivative Work based on material 
// that is copyright 2025 Kirk J Krauss and available at
//
//     https://developforperformance.com/MatchingWildcardsInGoSwiftAndCpp.html
// 
// Licensed under the Apache License, Version 2.0 (the "License"); you may not 
// use this file except in compliance with the License.  You may obtain a copy 
// of the License at
// 
//     https://www.apache.org/licenses/LICENSE-2.0
// 
// Unless required by applicable law or agreed to in writing, software 
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the 
// License for the specific language governing permissions and limitations 
// under the License.
//
#if !defined(__cplusplus)
#error "This is a demo program for the FastUtf8 C++ class."
#error "Testcases for the C-style functions in fastutf8.cpp are available."
#error "See testutf8.cpp and its documentation."
#endif

#if defined(_WIN32)
#include ‹windows.h›      // For SetConsoleOutputCP()
#endif  // _WIN32

#include ‹iostream›
#include ‹cstdio›
#include ‹cstring›
#include ‹string›
#include ‹cmath›
#include ‹chrono›
#include "fastutf8.h"
using namespace FastUtf8;

int UniseriespFindTokenDemo(void)
{
   // Define a pair of short Japanese buns and equivalent English sentences, 
   // plus a mix of the two.
   FastUtf8::Uniseries sJapanesePart1 = u8"これがミスター・ロボットに秘密を話すとこうなるんだ。 ";
   FastUtf8::Uniseries sEnglishPart1 = 
      u8"This is what happens when you tell Mr Robot a secret. ";
   FastUtf8::Uniseries sJapanesePart2 = u8"ミスター・ロボットがどうもありがとうと言っています。 ";
   FastUtf8::Uniseries sEnglishPart2 = u8"Mr Roboto says domo arigato.";
   FastUtf8::Uniseries sMixedPart2 = u8"Mr Roboto says どうもありがとうございます。";

   // Combine uniseries in the two languages.
   FastUtf8::Uniseries sJapaneseSentences = sJapanesePart1 + sJapanesePart2;
   FastUtf8::Uniseries sEnglishSentences = sEnglishPart1 + sEnglishPart2;
   FastUtf8::Uniseries sMixedSentences = sJapanesePart1 + sMixedPart2;

   // Define a delimiter set with both the Latin period (.) and Japanese kuten (。).
   FastUtf8::Uniseries sKutenSet = u8"。.";

   // Define a search pattern with a space followed by a wildcard.
   FastUtf8::Uniseries sSearchPattern = u8" *";

   // Extract the second portion of the combined Japanese bun.
   uint8_t *puzKuten = sJapaneseSentences.pFindToken(sKutenSet);
   uint8_t *puzFirst = sJapaneseSentences.pFindWild(sSearchPattern, &puzKuten);
   uint8_t *puzLast = sJapaneseSentences.pFindToken(puzFirst, sKutenSet);
   FastUtf8::Uniseries sJapanesePart2a(puzFirst, puzLast);

   std::cout ‹‹ std::endl ‹‹ "    From this combined content..." ‹‹ 
                      std::endl ‹‹ "        " ‹‹ sJapanesePart1 ‹‹ 
                      sJapanesePart2;
   std::cout ‹‹ std::endl ‹‹ "    ...extracted the second portion:" ‹‹ 
                      std::endl ‹‹"        " ‹‹ sJapanesePart2a ‹‹ std::endl;

   // Extract the second portion of the combined English sentences.
   puzKuten = sEnglishSentences.pFindToken(sKutenSet);
   puzFirst = sEnglishSentences.pFindWild(sSearchPattern, &puzKuten);
   puzLast = sEnglishSentences.pFindToken(puzFirst, sKutenSet);
   FastUtf8::Uniseries sEnglishPart2a(puzFirst, puzLast);

   std::cout ‹‹ std::endl ‹‹ "    From this combined content..." ‹‹ 
                      std::endl ‹‹ "        " ‹‹ sEnglishPart1 ‹‹ 
                      sEnglishPart2;
   std::cout ‹‹ std::endl ‹‹ "    ...extracted the second portion:" ‹‹ 
                      std::endl ‹‹"        " ‹‹ sEnglishPart2a ‹‹ std::endl;

   // Extract the second portion of the mixed combination.
   puzKuten = sMixedSentences.pFindToken(sKutenSet);
   puzFirst = sMixedSentences.pFindWild(sSearchPattern, &puzKuten);
   puzLast = sMixedSentences.pFindToken(puzFirst, sKutenSet);
   FastUtf8::Uniseries sMixedPart2a(puzFirst, puzLast);

   std::cout ‹‹ std::endl ‹‹ "    From this combined content..." ‹‹ 
                      std::endl ‹‹ "        " ‹‹ sMixedSentences;
   std::cout ‹‹ std::endl ‹‹ "    ...extracted the second portion:" ‹‹ 
                      std::endl ‹‹ "        " ‹‹ sMixedPart2a ‹‹ std::endl;

   return 0;
}

// Entry point for demo.
//
int main(void)
{
#if defined(_WIN32)
   SetConsoleOutputCP(CP_UTF8);
   SetConsoleCP(CP_UTF8);
#endif

   // Demo of search for any of a set of delimiters.
   std::cout ‹‹ "FastUtf8::Uniseries.pFindToken() demo" ‹‹ std::endl;
   UniseriespFindTokenDemo();
   return 0;
}

Uniseries::slice() methods

Range-Based Slice Methods

The index-driven range-based ::slice() method constructs a new Uniseries object from an existing one, making a deep copy of a portion of its content specified by the iFirst and iLast parameters.

Signature

Parameters

Discussion

---

The pointer-driven range-based ::slice() method constructs a new Uniseries object from an existing one, making a deep copy of a portion of its content specified by the pFirst and pLast parameters.  The method does no pointer validation.

Signature

Parameters

Discussion

---

Range-based constructors

Each of the ::fromSlice() methods constructs a new Uniseries object from a range of UTF-8 content.

Signature

Parameters

Discussion

---

Uniseries concatenation operators

Concatenation of raw buffer content

The pointer-driven concatenation operators reallocate the content buffer and perform a deep copy of the additional content.  The metadata is left as close as possible to the original metadata without falsifying it.  Any length limit is adjusted to accommodate the added content.

Signature

Parameter

Discussion

---

Concatenation of Uniseries content

The Uniseries-object-driven concatenation operators reallocate the content buffer and perform a deep copy of the additional content.  The metadata is left as close as possible to the original metadata without falsifying it.  Any length limit is adjusted to accommodate the added content.

Signature

Parameter

Discussion

---

Uniseries content separation methods

::pSeparate() methods returning objects

Each of these pointer-driven ::pSeparate() methods accepts a pointer to a buffer containing a tokenset and constructs a Uniseries object from a portion of the existing (this) object’s content.  The portion is derived based on a search for any of the delimiters in the set.  The new Uniseries object encompasses the content “ahead of” a found delimiter.  The existing this object is modified to encompass any remaining content ”after” the delimiter.  Its content is modified such that a delimiter is replaced with a null.

If the search can find no delimiter, the method effectively copies the content to the new object – but it’s not a deep copy; the new object and existing object will share the content buffer.  The method optionally trims white space from the new object’s content.

Signature

Parameters

Discussion

---

The Uniseries-object-driven tokenset-based ::pSeparate() method accepts a Uniseries-object containing a tokenset and constructs a Uniseries object from any first deimited portion found in the this object’s content.

Signature

Parameters

Discussion

---

The character-driven delimiter-based ::pSeparate() method accepts a single ASCII character as a delimiter and constructs a Uniseries object from any first deimited portion found in the this object’s content.

Signature

Parameters

---

::pSeparate() methods returning pointers

Pointer-returning ::pSeparate() overloads begin their tokenset search from an address within the this object’s content.  The address is specified via the first parameter.  The caller is responsible for ensuring that it refers within the content.

The address returned is the address of the first portion of delimited content.  The this content subsequently refers to any portion of the orginal content that remains, beyond the delimiter.  This method modifies the object’s content by replacing delimiters with nulls.

Signature

Parameters

Discussion

---

Example: ::caseCompare() and ::pSeparate() demo with ASCII text

The AsciiUniseriesMultiFuncDemo() function is part of the ::caseCompare() and ::pSeparate() demo.  Output from the "Content separation and trimming" portion of the demo program is shown in Fig. 5.

using namespace FastUtf8;

// Demonstrates this FastUtf8::Uniseries functionality for ASCII:
//
//   Fast one-call whole content case folding (lowercasing);
//   Case-insensitive whole content comparison;
//   Case-insensitive partial content comparison and find (returning an index);
//   Wildcard comparison, with / without case sensitivity;
//   Content separation over a buffer that remains in place (speedy!); and
//   Trimming of outboard white space.
//
int AsciiUniseriesMultiFuncDemo(void)
{
   // Declarations of multilingual variants of "Uniseries ASCII demo."
   Uniseries sDemo = "Uniseries ASCII demo";

   // Case-folding demo.
   std::cout ‹‹ "Case-folding (lowercasing) example:"  ‹‹ std::endl;

   // This first call is for UTF-8 compatibility.  Can use strlen() for ASCII, 
   // as an alternative.
   size_t sizeFolded = Uniseries::getSizeFolded(sDemo);
   Uniseries sDemoFolded1 = Uniseries::getFolded(sDemo, sizeFolded);
   Uniseries sDemoFolded2 = "uniseries ascii demo";

   if (sDemoFolded1 == sDemoFolded2)
   {
      std::cout ‹‹ "      Mixed case ASCII: " ‹‹ 
         sDemo ‹‹ std::endl;
      std::cout ‹‹ "     Case-folded ASCII: " ‹‹ 
         sDemoFolded1 ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-insensitive whole content comparison.
   std::cout ‹‹ "Case-insensitive whole content comparison:"  ‹‹ std::endl;

   if (sDemo.caseCompare(sDemoFolded1))
   {
      std::cout ‹‹ "     " ‹‹ sDemoFolded1 ‹‹ "  matches  " ‹‹ 
         sDemo ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-insensitive partial content comparison.
   std::cout ‹‹ "Case-insensitive partial content comparison:"  ‹‹ std::endl;

   Uniseries sAscii1 = sDemo + ", added text including MiXeD cAsE words";

   sAscii1.setCaseInsensitive();

   if (sAscii1.contains("mixed case"))
   {
      std::cout ‹‹ "     This content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ sAscii1 ‹‹ std::endl;
      std::cout ‹‹ "     ...contains the words..." ‹‹ std::endl;
	  std::cout ‹‹ "       mixed case" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-sensitive and case-insensitive partial content lookup.
   std::cout ‹‹ "Case-insensitive partial content lookup:"  ‹‹ std::endl;
   Uniseries sAscii2 = "This ASCII demo, with more text, has yet more";
   Uniseries sAscii3 = "The words, in multiple lines, are space-separated ";
   Uniseries sAscii4 = "Commas, too, can be separator characters";
   Uniseries sAsciiCombo = sAscii1 + "\n" + sAscii2 + "\n" + sAscii3 + 
      "\n" + sAscii4;

   // When we call ::find() on a Uniseries object that's set up for case-
   // sensitive comparison, we won't find an all-lowercase portion unless 
   // there's an all-lowercase match.
   std::cout ‹‹ "  -- Setting case sensitivity to true -- "  ‹‹ std::endl;
   sAsciiCombo.setCaseSensitivity(true);

   int iCommas1 = sAsciiCombo.find(sAscii4);
   int iCommas2 = sAsciiCombo.find("commas");

   if (iCommas1 > 0)
   {
      std::cout ‹‹ "     " ‹‹ sAscii4 ‹‹ "  has index  " ‹‹ 
         iCommas1 ‹‹ std::endl;
   }

   // Here, "Commas" fails to match "commas".
   if (iCommas2 > 0)
   {
      std::cout ‹‹ "     " ‹‹ "\"commas\"" ‹‹ "  has index  " ‹‹ 
         iCommas2 ‹‹ std::endl;
   }
   else
   {
      std::cout ‹‹ 
         "     Cannot find \"commas\"" ‹‹ 
         std::endl;
   }

   // By switching case sensitivity off, we change the default behavior for 
   // the comparison methods including ::find().
   std::cout ‹‹ "  -- Setting case sensitivity to false -- "  ‹‹ std::endl;
   sAsciiCombo.setCaseSensitivity(false);

   // Now that we've made that change, "Commas" matches "commas".
   iCommas2 = sAsciiCombo.find("commas");

   if (iCommas2 > 0)
   {
      std::cout ‹‹ "     " ‹‹ "\"commas\"" ‹‹ "  has index  " ‹‹ 
         iCommas2 ‹‹ std::endl;
   }
   else
   {
      std::cout ‹‹ 
         "     Cannot find \"commas\"" ‹‹ 
         std::endl;
   }

   std::cout ‹‹ std::endl;

   // Matching wildcards demo.
   std::cout ‹‹ "Matching wildcards:"  ‹‹ std::endl;

   if (sAsciiCombo.caseCompareWild("*ascii DEMO? WITH more*"))
   {
      std::cout ‹‹ 
         "     This content..." ‹‹ std::endl ‹‹ sAsciiCombo ‹‹ std::endl;
      std::cout ‹‹ 
         "     ...matches the wildcarded and inverse-cased sequence..." ‹‹ 
         std::endl;
	  std::cout ‹‹  "       *ascii DEMO? WITH more*" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of content separation and trimming.
   std::cout ‹‹ "Content separation and trimming:"  ‹‹ std::endl;

   // We'll extract "separated" from the four lines of text we've created.
   // First we duplicate the Uniseries object.  The pSeparate() calls modify 
   // the content by replacing separator delimiters with null terminators. 
   // The delimiters are the code points (in this case, ASCII characters) 
   // passed to Uniseries::pSeparate().
   Uniseries sComboDup = sAsciiCombo;

   // Separate the content by '-' characters.  For this demo, we capture 
   // but ignore the first portion.  (The compiler may warn us about that.)
   // The pSeparate() method relies on std::make_unique‹FastUtf8::Uniseries› 
   // to get the pointer that it returns.
   std::unique_ptr‹FastUtf8::Uniseries› psComboPart1 = 
                      sComboDup.pSeparate("-");

   // Separate the second portion by newline characters.
   std::unique_ptr‹FastUtf8::Uniseries› psComboPart2 = 
                      sComboDup.pSeparate("\n", /* bTrim = */ true);

   if (*psComboPart2 == "separated")  // note: without the trailing space
   {
      std::cout ‹‹ "     Extracted this trimmed content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ *psComboPart2 ‹‹ std::endl;
      std::cout ‹‹ "     ... from this content..." ‹‹ 
         std::endl ‹‹ sAsciiCombo ‹‹ std::endl;
      std::cout ‹‹ "     ... via two Uniseries::pSeparate() calls" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Another example.
   // We'll find a numeric sequence in text separated by spaces and lines.
   Uniseries s123 = "123";
   Uniseries sTwoSeparatorCombo = "abc def ghi\n123 456 789\nrst uvw xyz";

   // We duplicate the Uniseries object like we did previously.
   sComboDup = sTwoSeparatorCombo;

   // Separate the text by newline characters only, then separate the second 
   // portion by either newline or space characters.  The content buffers 
   // previously allocated for the reused objects get deallocated here.  The 
   // Uniseries objects otherwise remain allocated until they go out of scope.
   psComboPart1 = sComboDup.pSeparate("\n");
   psComboPart2 = sComboDup.pSeparate(" \n");

   if (*psComboPart2 == s123)
   {
      std::cout ‹‹ "     Extracted this content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ *psComboPart2 ‹‹ std::endl;
      std::cout ‹‹ "     ... from this content..." ‹‹ 
         std::endl ‹‹ sTwoSeparatorCombo ‹‹ std::endl;
      std::cout ‹‹ "     ... via two Uniseries::pSeparate() calls" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;
   return 0;
}

---

Further Uniseries tokenset search methods

::pFindToken() methods for raw buffer content

Each ::pFindToken() content comparison method searches the this object’s content for any delimiter within a set of delimiters, not modifying the content.  The tokenset search begins from an address, within the content, specified via the first parameter.  The caller is responsible for ensuring that the address is within the content.  If a delimiter is found, the method returns a pointer to the code point immediately prior to it.  Otherwise, the method returns nullptr.

Methods that accept a pointer to a delimiter buffer:

Signature

Parameters

---

Methods that accept a Uniseries object comprising the delimiter buffer:

Signature

Parameters

Discussion

---

::pFindToken() methods for Uniseries content

Single-parameter overloads of pFindToken() begin their search at the top of the this content.

Methods that accept a pointer to a delimiter buffer:

Signature

Parameter

---

Method that accepts a Uniseries object comprising the delimiter buffer:

Signature

Parameter

Discussion

---

Uniseries iterator

The methods within the iterator class apply to individual code points within the content buffer of a Uniseries object.

The iterator constructor provides for a buffer comprising a range of contiguous code points.

Signature

Discussion

---

Basic dereference operators include operator* and operator->.

Signature

Discussion

---

The prefix and postfix increment operators each advance the iterator by a code point.

Signature

---

The prefix and postfix decrement operators each backtrack the iterator by a code point.

Signature

---

These are equality operators for individual code points in content.

Signature

---

The begin() iterator references the first code point in Uniseries content.  The end() iterator refers immediately past the last code point.

Signature

---

Example: Iterator demo

The UniseriesIteratorDemo() and ContainsJapanese() functions are part of the iterator demo (IteratorDemo.cpp).  Output is shown in Fig. 3.

// Returns true if the query content includes Japanese code points, and false 
// if it doesn't.
bool ContainsJapanese(FastUtf8::Uniseries sQuery)
{
   bool bRetVal = false;

   for (FastUtf8::Uniseries::Iterator itr = sQuery.begin(); 
                      itr != sQuery.end(); ++itr)
   {
      uint32_t n = *itr;                       // UTF-8 code point in sQuery

      if ((n >= 0xE38180 && n ‹= 0xE383BF) ||  // Hiragana/Katakana range
          (n >= 0xE4B880 && n ‹= 0xE9BFBF) ||  // CJK unified ideographs
          (n >= 0xEFBDA5 && n ‹= 0xEFBE9F))    // Shift-JIS encodings
      {
         bRetVal = true;
      }
   }

   return bRetVal;
}

int UniseriesIteratorDemo(void)
{
   FastUtf8::Uniseries sEnglish = u8"Mr. Roboto says domo arigato.";
   FastUtf8::Uniseries sMixed = u8"Mr. Roboto says どうもありがとうございます。";

   if (!ContainsJapanese(sEnglish))
   {
      std::cout ‹‹ std::endl ‹‹ "    This content..." ‹‹ std::endl ‹‹ 
                      "        " ‹‹ sEnglish;
      std::cout ‹‹ std::endl ‹‹ "    ...includes no Japanese code points." ‹‹ 
                      std::endl;
   }

   if (ContainsJapanese(sMixed))
   {
      std::cout ‹‹ std::endl ‹‹ "    This content..." ‹‹ std::endl ‹‹ 
                      "        " ‹‹ sMixed;
      std::cout ‹‹ std::endl ‹‹ "    ...includes Japanese code points." ‹‹ 
                      std::endl;
   }

   return 0;
}

Uniseries get and set methods

::getContent() methods

These are basic getter / setter methods for whole content and its metadata.  This first one gets a pointer to the this object’s content buffer.  The static methods demo has example code.

Signature

---

These ::getContent() methods make a deep copy of the whole content.  The developer is responsible for ensuring that the buffer receiving the content is sufficient.

Signature

---

::getMetadata() method

This method returns a uint64_t value comprising the this object’s metadata.

Signature

---

::is7Bit() methods

Is the inbound content all 7-bit ASCII characters?  If so, the ::is7Bit() method sets the IS_7BIT_CHAR_STRING flag for this and returns true.  Otherwise, the method clears that flag and returns false.  To get its result, it calls the Is7BitUtf8() function.  The static methods demo has example code that invokes ::is7Bit().

Signature

---

Methods for setting case sensitivity

The IS_CASE_INSENSITIVE flag for this can be set via any of the next methods.

Signature

---

Methods for setting the length limit

The length limit for this can be set via any of the next methods. Selecting a nonzero length limit also sets the IS_LENGTH_LIMITEDflag.

Signature

---

::getLength() methods

Given a byte count, this method returns the corresponding count of code points between the beginning of the this object’s content and the last complete code point encompassing the given number of bytes.  To get its result, it calls the LenSizeOfUtf8() function.  The ::getLength() methods perform no UTF-8 validation other than null checking. .

Signature

---

This method returns the object’s current content length as a count of its code points.  To get its result, it calls the CodePointCountUtf8() function.

Signature

---

These static ::getLength() methods return the length of the inbound content as a count of its code points, which they get by calling the CodePointCountUtf8() function.  The static methods demo has example code.

Signature

---

::getSize() methods

This ::getSize() method returns the this object’s content’s size in bytes.  The ::getSize() methods get their findings by calling one or another of the SizeOf*Utf8() functions.  They perform no UTF-8 validation other than null checking.

Signature

---

These static ::getSize() implementations return the size of the inbound content, in bytes.  The static methods demo has example code.

Signature

---

Uniseries case folding methods

::getSizeFolded() methods

The static ::getSizeFolded() methods returns the number of bytes needed to store the inbound content after case folding. This size may be larger or smaller than the unfolded size. The nonstatic method determines the folded size for the this object. These methods call one or another of theSizeOfFolded*Utf8() functions.  The static methods demo has example code that invokes ::getSizeFolded().

Signature

---

::getFolded() methods

Each ::getFolded() method makes a deep copy of the whole content after case folding.  The developer is responsible for ensuring that the buffer receiving the content is sufficient.  The size needed for folded content may be greater or less than the original content’s size, in bytes.  To predetermine the needed size, invoke the getSizeFolded() method (described immediately above).

Signature

Discussion

---

Example: Static methods demo

The UniseriesStaticMethodsDemo() function is part of the static methods demo (StaticMethodsDemo.cpp).  The demo program’s output is shown in Fig. 3.

// Demonstrates the static FastUtf8::Uniseries methods:
//
//   getLength()      Get content length in code points.
//   getSize()        Get content size in bytes.
//   getSizeFolded()  Get content size anticipated after case folding.
//   getFolded()      Get case-folded content.
//   is7Bit()         Is the current content all 7-bit ASCII characters?
//
// The static methods are provided for ease of access without requiring a 
// Uniseries instance.
//
int UniseriesStaticMethodsDemo(void)
{
   // Declare an ASCII string and a circled Latin series.
   char szBerryQuote[88] = 
    "I don't remember when I did not know Port William, the town and the neighborhood.";
   uint8_t uzCircledLatinQuote[353] = 
    u8"Ⓘ ⓓⓞⓝ’ⓣ ⓡⓔⓜⓔⓜⓑⓔⓡ ⓦⓗⓔⓝ Ⓘ ⓓⓘⓓ ⓝⓞⓣ ⓚⓝⓞⓦ Ⓟⓞⓡⓣ Ⓦⓘⓛⓛⓘⓐⓜ, ⓣⓗⓔ ⓣⓞⓦⓝ ⓐⓝⓓ ⓣⓗⓔ ⓝⓔⓘⓖⓗⓑⓞⓡⓗⓞⓞⓓ.";

   // Demonstrate methods that get the length and size of each.
   std::cout ‹‹ "Length and size example:"  ‹‹ std::endl;

   int     lenBerry = FastUtf8::Uniseries::getLength(szBerryQuote);
   int     lenCircled = FastUtf8::Uniseries::getLength(uzCircledLatinQuote);
   size_t  sizeBerry = FastUtf8::Uniseries::getSize(szBerryQuote);
   size_t  sizeCircled = FastUtf8::Uniseries::getSize(uzCircledLatinQuote);

   std::cout ‹‹ "ASCII quote:"  ‹‹ std::endl ‹‹ szBerryQuote ‹‹ std::endl;
   std::cout ‹‹ "     Length: " ‹‹ lenBerry ‹‹ std::endl;
   std::cout ‹‹ "      Bytes: " ‹‹ sizeBerry ‹‹ std::endl ‹‹ std::endl;
   std::cout ‹‹ "In circled Latin:"  ‹‹ std::endl ‹‹ uzCircledLatinQuote ‹‹ 
                   std::endl;
   std::cout ‹‹ "     Length: " ‹‹ lenCircled ‹‹ std::endl;
   std::cout ‹‹ "      Bytes: " ‹‹ sizeCircled ‹‹ std::endl ‹‹ std::endl;

   // Allocate buffers sufficient for the case-folded ASCII and circled Latin 
   // content.
   size_t  sizeFoldedBerry = FastUtf8::Uniseries::getSizeFolded(szBerryQuote);
   size_t  sizeFoldedCircled = FastUtf8::Uniseries::getSizeFolded(
                   uzCircledLatinQuote);
   char    *pszFoldedBerry = 
                   reinterpret_cast‹char *› (std::malloc(sizeFoldedBerry));
   uint8_t *puzFoldedCircled = 
                   reinterpret_cast‹uint8_t *› (std::malloc(sizeFoldedCircled));

   // Place the case-folded ASCII and circled Latin content into the buffers.
   if (pszFoldedBerry && puzFoldedCircled)
   {
      Uniseries sFoldedBerry = FastUtf8::Uniseries::getFolded(szBerryQuote);
      Uniseries sFoldedCircled = FastUtf8::Uniseries::getFolded(
                   uzCircledLatinQuote);

      strncpy(pszFoldedBerry, 
              reinterpret_cast‹char *› (sFoldedBerry.getContent()), 
              sizeFoldedBerry);
      strncpy(reinterpret_cast‹char *› (puzFoldedCircled), 
              reinterpret_cast‹char *› (sFoldedCircled.getContent()), 
              sizeFoldedCircled);

      // Show the case-folded content, and demonstrate 7-bit ASCII check.
      std::cout ‹‹ "Case-folded ASCII quote:"  ‹‹ std::endl ‹‹ 
                   pszFoldedBerry ‹‹ std::endl;

      if (FastUtf8::Uniseries::is7Bit(pszFoldedBerry))
      {
         std::cout ‹‹ "     This is 7-bit ASCII text."  ‹‹ std::endl;
      }
      else
      {
         std::cout ‹‹ "     This is not 7-bit ASCII text."  ‹‹ std::endl;
      }
      
      std::cout ‹‹ std::endl ‹‹ 
                   "Case-folded circled Latin quote:"  ‹‹ std::endl ‹‹ 
                   puzFoldedCircled ‹‹ std::endl;

      if (FastUtf8::Uniseries::is7Bit(puzFoldedCircled))
      {
         std::cout ‹‹ "     This is 7-bit ASCII text."  ‹‹ std::endl;
      }
      else
      {
         std::cout ‹‹ "     This is not 7-bit ASCII text."  ‹‹ std::endl;
      }
      
      free(pszFoldedBerry);
      free(puzFoldedCircled);
   }
   
   return 0;
}

---

Whole Uniseries comparison operators and methods

Equality operators

These are equality operators for whole content comparison.  They call one or another of the *CompareUtf8() functions or str*cmp() functions.  The ::caseCompare() and ::pSeparate() demo with ASCII text has example code that invokes operator==.

Signature

---

::caseCompare() methods

The caseCompare() method is identical to the equality operator but strictly case-insensitive; it doesn’t check the IS_CASE_INSENSITIVE flag.  It calls one or another of the *CaseCompareUtf8() functions or str*casecmp() functions.  The ::caseCompare() and ::pSeparate() demo with ASCII text has example code that invokes ::caseCompare().

Signature

---

Comparison operator for pointer == object

This Uniseries comparison operator applies for pointer == object.

Signature

---

Comparison pperator for object == pointer

This Uniseries comparison operator applies for object == pointer.

Signature

---

Uniseries() partial content comparison methods

::contains() methods for raw buffer content

The ::contains() partial content comparison methods for inbound buffer content return true if the this object’s content contains the inbound content, and false otherwise.

Signature

Parameter

Discussion

---

This ASCII-character-driven ::contains() partial content comparison method returns true if the this object’s content contains the inbound ASCII character, and false otherwise.

Signature

Parameter

Discussion

---

::contains() method for Uniseries content

This partial content comparison method returns true if the this object‘s content contains the that object‘s content, and false otherwise.

Signature

Discussion

---

::find() methods for raw buffer content

The ::find() partial content comparison method for inbound buffer content returns an index of the inbound content within the this object’s content – that is, a count of the code points between the beginning of this content and any first match – or a negative return value (-1) in case that content isn’t found.

Signature

Discussion

---

This ASCII-character-driven ::find() partial content comparison method returns an index of the inbound ASCII character within the this object’s content – that is, a count of the code points between the beginning of this content and any first match – or a negative return value (-1) in case the character is not found.

Signature

Discussion

---

::find() method for Uniseries content

This partial content comparison method returns an index of the that object’s content within the this object’s content – that is, a count of the code points between the beginning of this content and any first match – or a negative return value (-1) in case that content is not found.

Signature

Discussion

---

Example: ::find() demo with UTF-8 content

The Utf8UniseriesDemo() function is part of the fundamentals demo (FundamentalsDemo.cpp).  Output from the demo program is shown in Fig. 2.

// This code is similar to the above AsciiUniseriesDemo() code but with 
// Japanese content.
//
int Utf8UniseriesDemo(void)
{
   // 1. Creation and Concatenation
   Uniseries s1 = "こんにちは";
   Uniseries s2 = "は世界";
   // Using + operator
   Uniseries s3 = s1 + "、" + s2 + "！"; // "Hello, World!"
   s3 = s3 + " いらっしゃいませ。"; // "Hello, World! Welcome."

   std::cout << "Concatenated (連結済み)： " << s3 << std::endl;

   // 2. Substring (Position, Length)
   // Extract "World"
   Uniseries sub = s3.slice(/* iFirst = */ 7, /* iLast = */ 12);
   std::cout << "Substring (部分文字列)： " << sub << std::endl;

   // 3. Substring Find
   int iFound = s3.find("いらっしゃいませ");

   if (iFound != -1)
   {
      std::cout << "\"いらっしゃいませ\" found at (発見場所)： " << iFound << std::endl;
   }

   // 4. Whole String Comparison
   Uniseries s4 = "こんにちは、は世界！ いらっしゃいませ。";

   if (s3 == s4)
   {
      std::cout << "Strings are equal (文字列は等しい)。" << std::endl;
   }

   // 5. Separate (Splitting by code point)
   int iCommaPos = s3.find("、");

   if (iCommaPos != -1)
   {
      Uniseries sFirstPart = s3.slice(0, iCommaPos);
      Uniseries sSecondPart = s3.slice(iCommaPos + 2); // Skip ", "
      std::cout << "Separated (分離済み)： \"" << sFirstPart << "\" and \"" << 
         sSecondPart << "\"" << std::endl;
   }

   return 0;
}

---

::pFind() methods for raw buffer content

The ::pFind() partial content comparison methods for inbound buffer content return a pointer to the location of a match for the inbound content within the this object’s content – that is, a pointer to the beginning of any first match within this – or a nullptr return value in case the content is not found.  The optional pFirst parameter refers to a location, in the this content, to begin seeking a match.  Each method updates the optional *ppLast parameter to return the ending location of any first match within this.

Signature

Parameters

Discussion

---

The ASCII-character-driven ::pFind() partial content comparison method returns a pointer to the location of the inbound character within the this object’s content – that is, a pointer to the beginning of any first match within this – or a nullptr return value in case that content is not found.  The optional pFirst parameter refers to a location, in the this content, to begin seeking a match.  The method updates the optional *ppLast parameter to return the ending location of any first match within this.

Signature

Parameters

Discussion

---

::pFind() method for Uniseries content

This pFind() partial content comparison method returns a pointer to any first occurrence of the that object’s content within the this object’s content, or nullptr in case that content is not found.

Signature

Parameters

Discussion

---

::caseContains() methods for raw buffer content

The ::caseContains() case-insensitive partial content comparison methods for inbound buffer content return true if the this object’s content contains the inbound content, and false otherwise.  It disregards the IS_CASE_INSENSITIVE flag.

Signature

Parameter

Discussion

---

::caseContains() method for Uniseries content

This case-insensitive partial content comparison method returns true if the this object‘s content contains the that object‘s content, and false otherwise.  It disregards the IS_CASE_INSENSITIVE flag.

Signature

Discussion

---

::caseFind() methods for raw buffer content

The ::caseFind() case-insensitive partial content comparison method for inbound buffer content returns an index of the inbound content within the this object’s content – that is, a count of the code points between the beginning of this content and any first match – or a negative return value (-1) in case that content is not found.  It disregards the IS_CASE_INSENSITIVE flag.

Signature

Discussion

---

::caseFind() method for Uniseries content

This case-insensitive partial content comparison method returns an index of the that object’s content within the this object’s content – that is, a count of the code points between the beginning of this content and any first match – or a negative return value (-1) in case that content is not found.  It disregards the IS_CASE_INSENSITIVE flag.

Signature

Discussion

---

::casepFind() methods for raw buffer content

The ::casepFind() case-insensitive partial content comparison methods for inbound buffer content return a pointer to the location of a match on the inbound content within the this object’s content – that is, a pointer to the beginning of any first match within this – or a nullptr return value in case that content is not found.  The optional pFirst parameter refers to a location, in the this content, to begin seeking a match.  Each method updates the optional *ppLast parameter to return the ending location of any first match within this.  These methods disregard the IS_CASE_INSENSITIVE flag.

Signature

Parameters

Discussion

---

::casepFind() method for Uniseries content

This casepFind() case-insensitive partial content comparison method returns a pointer to any first occurrence of the that object’s content within the this object’s content, or nullptr in case that content is not found.  It disregards the IS_CASE_INSENSITIVE flag.

Signature

Parameters

Discussion

---

Example: ::pFind() demo with ASCII content

The AsciiUniseriespFindMute() function is invoked for timing comparisons as explained in the FastUtf8 overview.  The function is part of the fundamentals demo (FundamentalsDemo.cpp).  Output showing timing comparisons is shown in the overview’s Fig. 8.

int AsciiUniseriespFindMute(void)
{
   // 1. Creation and Concatenation
   Uniseries s1 = "Hello";
   Uniseries s2 = "World";
   // Using + operator
   Uniseries s3 = s1 + ", " + s2 + "!"; // "Hello, World!"
   s3 = s3 + " Welcome."; // "Hello, World! Welcome."

   // 2. Substring (Position, Length)
   // Extract "World! Welcome."
   Uniseries sub = s3.slice(/* iFirst = */ 7, /* iLast = */ 22);

   if (sub == s1)
   {
      return 1;
   }

   // 3. Substring Find
   uint8_t *pFound = s3.pFind("Welcome");

   if (!pFound)
   {
      return 1;
   }

   // 4. Whole String Comparison
   Uniseries s4 = "Hello, World! Welcome.";

   if (s3 != s4)
   {
      return 1;
   }

   // 5. Separate (Splitting by character)
   uint8_t *pFirstPart = s3.pFind(',');

   if (!std::strcmp(
         reinterpret_cast‹char *› (const_cast‹uint8_t *› (pFirstPart)), 
         reinterpret_cast‹char *› (const_cast‹uint8_t *› (s3.getContent()))))
   {
      return 1;   // "Hello" != " World! Welcome."
   }

   return 0;
}

Uniseries() methods for matching wildcards

::wildCompare() methods

The ::wildCompare() methods provide for wildcard-based content comparison.  The this content is the content that may include the ‘*’ or ‘?’ wildcards.

Signature

Parameter

Discussion

---

::wildCaseCompare() methods

The ::wildCaseCompare() methods provide for strictly case-insensitive wildcard-based content comparison.  They don’t check the IS_CASE_INSENSITIVE flag.  The this content is the content that may include the ‘*’ or ‘?’ wildcards.

Signature

Discussion

---

::compareWild() methods

The ::compareWild() methods provide for wildcard-based content comparison.  The inbound content is the content that may include the ‘*’ or ‘?’ wildcards.

Signature

Parameter

Discussion

---

::caseCompareWild() methods

The ::caseCompareWild() methods provide for strictly case-insensitive wildcard-based content comparison.  They don’t check the IS_CASE_INSENSITIVE flag.  The inbound content is the content that may include the ‘*’ or ‘?’ wildcards.

Signature

Discussion

Example: ::caseCompare() and ::pSeparate() demo with UTF-8 content

The Utf8UniseriesMultiFuncDemo() function is part of the ::caseCompare() and ::pSeparate() demo (CaseCompareAndSeparateDemo.cpp).  Output for the portion of the demo program that handles ASCII content is shown in Figs. 4 and 5.  Output for the portion that handles multi-byte UTF-8 content, as coded in Utf8UniseriesMultiFuncDemo(), is shown in Figs. 6 and 7.

// Demonstrates this FastUtf8::Uniseries functionality for UTF-8:
//
//   UTF-8 support;
//   Easy-to-use case folding;
//   Case-insensitive whole content comparison;
//   Case-insensitive partial content comparison and find (returning an index);
//   Wildcard comparison, with / without case sensitivity;
//   Content separation over a buffer that remains in place; and
//   Trimming of outboard white space;
//   Validations:
//     that content comprises 7-bit ASCII characters, or
//     that content comprises valid UTF-8 code points.
//
// This function demonstrates those aspects of FastUtf8.
//
int Utf8UniseriesMultiFuncDemo(void)
{
   // Declarations of multilingual variants of "This is a UTF-8 demo."
   Uniseries sAdlam = "𞤚𞤸𞤭𞤧 𞤭𞤧 𞤢 𞤓𞤚𞤊-𞥘 𞤣𞤫𞤥𞤮";
   Uniseries sAmharic = "ይህ የ UTF-8 ማሳያ ነው።";
   Uniseries sArmenian = "Սա UTF-8 դեմո է։";
   Uniseries sBangla = "এটি UTF-8 এর একটি ডেমো।";
   Uniseries sCantonese = "呢個系UTF-8嘅演示。";
   Uniseries sCherokee = "ᎯᎠ ᎤᏣᏔ-8 ᏗᎦᏙᎤᏍᏗ ᎠᏍᎦᏯ.";
   Uniseries sDeseret = "𐐜𐐮𐑅 𐐮𐑆 𐐩 UTF-8 𐐼𐐯𐑋𐐬.";
   Uniseries sEnglish = "This is a UTF-8 demo.";
   Uniseries sGreek = "Αυτό είναι ένα demo UTF-8.";
   Uniseries sHebrew = "זהו הדגמה של UTF-8.";
   Uniseries sHindi = "यह एक UTF-8 डेमो है।";
   Uniseries sInuktitut = "ᐅᓇ UTF-8 ᑕᑯᒃᓴᐅᑎᑕᐅᔪᖅ.";
   Uniseries sJapanese = "これはUTF-8のデモです。";
   Uniseries sKlingon = "   -8 .";
   Uniseries sNepali = "यो UTF-8 को डेमो हो।";
   Uniseries sOdia = "ଏହା UTF-8 ର ଏକ ଡେମୋ ।";
   Uniseries sRunic = "ᛏᚺᛁᛊ ᛁᛊ ᚨ ᚢᛏᚠ-ᚹ ᛞᛖᛗᛟ·";
   Uniseries sRussian = "Это демонстрация UTF-8.";
   Uniseries sPashto = "دا د UTF-8 یوه نمونه ده.";
   Uniseries sPersian = "این یک دمو از UTF-8 است.";
   Uniseries sTamil = "இது UTF-8 இன் டெமோ ஆகும்.";
   Uniseries sTigrinya = "እዚ ናይ UTF-8 ዲሞ እዩ።";
   Uniseries sTelugu = "นఇది UTF-8 యొక్క డెమో.";
   Uniseries sThai = "นี่คือการสาธิตของ UTF-8";
   Uniseries sUrdu = "یہ UTF-8 کا ایک ڈیمو ہے۔";

   // Case-folding demo.
   std::cout ‹‹ "Case-folding examples:"  ‹‹ std::endl;

   size_t sizeFolded = Uniseries::getSizeFolded(sAdlam);
   Uniseries sAdlamFolded1 = Uniseries::getFolded(sAdlam, sizeFolded);
   Uniseries sAdlamFolded2 = "𞤼𞤸𞤭𞤧 𞤭𞤧 𞤢 𞤵𞤼𞤬-𞥘 𞤣𞤫𞤥𞤮";

   if (sAdlamFolded1 == sAdlamFolded2)
   {
      std::cout ‹‹ "      Mixed case Adlam script: " ‹‹ 
         sAdlam ‹‹ std::endl;
      std::cout ‹‹ "     Case-folded Adlam script: " ‹‹ 
         sAdlamFolded1 ‹‹ std::endl;
   }

   sizeFolded = Uniseries::getSizeFolded(sEnglish);
   Uniseries sEnglishFolded1 = Uniseries::getFolded(sEnglish, sizeFolded);
   Uniseries sEnglishFolded2 = "this is a utf-8 demo.";

   if (sEnglishFolded1 == sEnglishFolded2)
   {
      std::cout ‹‹ "      Mixed case English text: " ‹‹ 
         sEnglish ‹‹ std::endl;
      std::cout ‹‹ "     Case-folded English text: " ‹‹ 
         sEnglishFolded1 ‹‹ std::endl;
   }

   sizeFolded = Uniseries::getSizeFolded(sGreek);
   Uniseries sGreekFolded1 = Uniseries::getFolded(sGreek, sizeFolded);
   Uniseries sGreekFolded2 = "αυτό είναι ένα demo utf-8.";

   if (sGreekFolded1 == sGreekFolded2)
   {
      std::cout ‹‹ "      Mixed-case Greek stichos: " ‹‹ 
         sGreek ‹‹ std::endl;
      std::cout ‹‹ "     Case-folded Greek stichos: " ‹‹ 
         sGreekFolded1 ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-insensitive whole content comparison.
   std::cout ‹‹ "Case-insensitive whole content comparison:"  ‹‹ std::endl;
   sEnglish.setCaseSensitivity(/* bCaseSensitive = */ false);
   sGreek.setCaseInsensitive();

   if (sAdlam.caseCompare(sAdlamFolded1))
   {
      std::cout ‹‹ "     " ‹‹ sAdlamFolded1 ‹‹ "  matches  " ‹‹ 
         sAdlam ‹‹ std::endl;
   }

   if (sEnglishFolded1 == "this is a utf-8 demo.")
   {
      std::cout ‹‹ "     " ‹‹ sEnglishFolded1 ‹‹ "  matches  " ‹‹ 
         "This is a UTF-8 demo." ‹‹ std::endl;
   }

   if (sGreekFolded1 == sGreek)
   {
      std::cout ‹‹ "     " ‹‹ sEnglishFolded1 ‹‹ "  matches  " ‹‹ 
         sEnglish ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-insensitive partial content comparison.
   std::cout ‹‹ "Case-insensitive partial content comparison:"  ‹‹ std::endl;

   Uniseries sMulti1 = sHindi + ", " + sRussian + ", " + 
      sPashto + ", " + sPersian + ", " + sTamil + ", " + sTigrinya;

   sMulti1.setCaseInsensitive();

   if (sMulti1.contains("это демонстрация utf-8"))
   {
      std::cout ‹‹ "     This content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ sMulti1 ‹‹ std::endl;
      std::cout ‹‹ "     ...contains the Russian stroka..." ‹‹ std::endl;
	  std::cout ‹‹ "       это демонстрация utf-8" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of case-sensitive and case-insensitive partial content lookup.
   std::cout ‹‹ "Case-insensitive partial content lookup:"  ‹‹ std::endl;
   Uniseries sMulti2 = sAmharic + ", " + sBangla + ", " + 
      sCantonese + ", " + sCherokee + ", " + sDeseret + ", " + sHebrew;
   Uniseries sMulti3 = sRunic + ", " + sInuktitut + ", " + 
      sJapanese + ", " + sKlingon + ", " + sNepali + ", " + sOdia;
   Uniseries sMulti4 = sTelugu + ", " + sThai + ", " + sUrdu;
   Uniseries sMultiCombo = sMulti1 + "\n" + sMulti2 + "\n" + sMulti3 + 
      "\n" + sMulti4;

   std::cout ‹‹ "  -- Setting case sensitivity to true -- "  ‹‹ std::endl;
   sMultiCombo.setCaseSensitivity(true);

   int iOdiaLain = sMultiCombo.find(sOdia);
   int iBindrune = sMultiCombo.find("ᛁᛊ");

   if (iOdiaLain > 0)
   {
      std::cout ‹‹ "     " ‹‹ sOdia ‹‹ "  has index  " ‹‹ 
         iOdiaLain ‹‹ std::endl;
   }

   if (iBindrune > 0)
   {
      std::cout ‹‹ "     " ‹‹ "ᛁᛊ" ‹‹ "  has index  " ‹‹ 
         iBindrune ‹‹ std::endl;
   }

   int iRussianStroka = sMultiCombo.find("это демонстрация utf-8");

   if (iRussianStroka > 0)
   {
      std::cout ‹‹ "     " ‹‹ "это демонстрация utf-8" ‹‹ "  has index  " ‹‹ 
         iRussianStroka ‹‹ std::endl;
   }
   else
   {
      std::cout ‹‹ 
         "     Cannot find Russian stroka / Не могу найти русскую строку" ‹‹ 
         std::endl;
   }

   std::cout ‹‹ "  -- Setting case sensitivity to false -- "  ‹‹ std::endl;
   sMultiCombo.setCaseSensitivity(false);
   iRussianStroka = sMultiCombo.find("это демонстрация utf-8");

   if (iRussianStroka > 0)
   {
      std::cout ‹‹ "     " ‹‹ "это демонстрация utf-8" ‹‹ "  has index  " ‹‹ 
         iRussianStroka ‹‹ std::endl;
   }
   else
   {
      std::cout ‹‹ 
         "     Cannot find Russian stroka / Не могу найти русскую строку" ‹‹ 
         std::endl;
   }

   std::cout ‹‹ std::endl;

   // Matching wildcards demo.
   std::cout ‹‹ "Matching wildcards:"  ‹‹ std::endl;

   if (sMulti1.caseCompareWild("*ДЕМОНСТРА?ИЯ utf-8*"))
   {
      std::cout ‹‹ "     This content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ sMulti1 ‹‹ std::endl;
      std::cout ‹‹ "     ...matches the wildcarded and inverse-cased Cyrillic / Latin sequence..." ‹‹ std::endl;
	  std::cout ‹‹  "       *ДЕМОНСТРА?ИЯ utf-8*" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of content separation and trimming.
   std::cout ‹‹ "Content separation and trimming:"  ‹‹ std::endl;

   Uniseries sMultiComboDup = sMultiCombo;
   std::unique_ptr‹FastUtf8::Uniseries> psMultiComboPart1 = 
                      sMultiComboDup.pSeparate("\n");
   std::unique_ptr‹FastUtf8::Uniseries> psMultiComboPart2 = 
                      sMultiComboDup.pSeparate(" \n", /* bTrim = */ true);

   if (sAmharic == *psMultiComboPart2)
   {
      std::cout ‹‹ "     Extracted this content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ *psMultiComboPart2 ‹‹ std::endl;
      std::cout ‹‹ "     ... from this content..." ‹‹ 
         std::endl ‹‹ sMultiCombo ‹‹ std::endl;
      std::cout ‹‹ "     ... via two Uniseries::pSeparate() calls" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   // Demo of 7-bit character validation and of converting 8-bit ASCII to 
   // valid UTF-8.
   std::cout ‹‹ "Validations:"  ‹‹ std::endl;
   char sz8BitAscii[6] = "\x80\x81\x82\xA5\xEA"; 
   Uniseries sFormer8Bit = sz8BitAscii;

   if (sEnglish.is7Bit())
   {
      std::cout ‹‹ "     This content...  " ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ sEnglish ‹‹ std::endl;
      std::cout ‹‹ "     ...is 7-bit ASCII text" ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;

   if (sFormer8Bit.validate())
   {
      std::cout ‹‹ "     This 8-bit ASCII content...  " ‹‹ std::endl;
#if defined(_WIN32)
SetConsoleOutputCP(437);
#endif
      std::cout ‹‹ "        " ‹‹ sz8BitAscii ‹‹ std::endl;
#if defined(_WIN32)
   SetConsoleOutputCP(CP_UTF8);
#endif
      std::cout ‹‹ "     ...has been converted to this UTF-8 content..." ‹‹ std::endl;
      std::cout ‹‹ "        " ‹‹ sFormer8Bit ‹‹ std::endl;
   }

   std::cout ‹‹ std::endl;
   return 0;
}

Uniseries() methods for targeted wildcard search

The targeted wildcard search concept is described with introductory comments and graphical examples as part of the FastUtf8 overview.  There’s also a discussion of the design of the ::pFindWild() and ::casepFindWild() methods for targeted wildcard search, together with an outline of a use case for handling user input.  Usage of those methods is explained here.  They operate by calling one or another of the Wild*FindUtf8() functions for targeted wildcard search.

::pFindWild() methods

Each of the ::pFindWild() partial content comparison methods scans the this content for a first match on the inbound search pattern, which is expected to contain wildcards, and returns a pointer to the location within the the content corresponding to any first matching ‘*’ wildcard.  If no match is found, or if no first ‘*’ wildcard is found, these methods set *ppFirst, *ppLast, and *ppTarget to nullptr and return nullptr.

Signature

Parameters

Discussion

---

::casepFindWild() methods

Case-insensitive implementation of the ::pFindWild() partial content comparison method.

Signature

Discussion

---

Example: Targeted wildcard search demo

The FetchAuthorByTitleUsingFastUtf8Uniseries() function in the FastUtf8 overview has example code that invokes ::casepFindWild().  The function is part of the semistructured search demo (SemistructuredSearchDemo.cpp).  Part of the demo program’s output is shown in Fig. 8.

Uniseries methods for slices within slices

These slice search methods are intended for use on relatively large UTF-8 datasets, such as loaded machine-generated JSON or XML files, that may create enough memory pressure to be accessed in slices so as to allow the operating system to page portions of them to disk.  They’re implemented as interfaces for functions that support a wildcard-driven query capability for hierarchical semistructured data, similar to the capability of a relational database given a basic SQL SELECT query.  In addition to that support for queries, these methods include content lookup with and without wildcards, as well as a form of targeted wildcard search that returns vectorized matches for every ‘*’ wildcard in a search pattern.  The functions invoked by these methods are variations on the legacy C compatible functions for targeted wildcard search.  Because they include variations that depend on std::array and std::set containers, they have no legacy C counterparts.  All of the slice search functionality described in this section is implemented for C++ only.

---

::pFindSlice() methods for raw buffer content

Each of these ::pFindSlice() methods returns a pointer to the first occurrence of the inbound search pattern slice within a slice of the this content, or nullptr in case no such search pattern slice is found.  The search pattern slice comprises the bytes between the locations indicated by pPatternBegin and pPatternEnd.  The optional pBegin parameter refers to a location, in the this content, to begin seeking a match.  If the optional ppEnd parameter is not nullptr, the method disregards the IS_LENGTH_LIMITED flag and either stops searching at the location specified via ppEnd or updates ppEnd to provide the limit of any first matching content within this.

Signature

Parameters

Discussion

---

::pFindSlice() method for Uniseries content

This ::pFindSlice() method returns a pointer to any first match on sPattern between the locations indicated by the optional pBegin and ppEnd parameters.  If those parameters are nullptr, returns a pointer to any first match on sPattern within this.  Otherwise behaves like the ::pFindSlice() methods described above.  Returns nullptr in case the slice is not found.  The method ignores the IS_LENGTH_LIMITED flag, unless the ppEnd parameter is nullptr.

Signature

Parameters

---

::casepFindSlice() methods for raw buffer content

Each of these ::casepFindSlice() methods performs a case-insensitive search and returns a pointer to the first occurrence of the inbound search pattern slice within a slice of the this content, or nullptr in case no such search pattern slice is found.  The search pattern slice comprises the bytes between the locations indicated by pPatternBegin and pPatternEnd.  The optional pBegin parameter refers to a location, in the this content, to begin seeking a match.  If the optional ppEnd parameter is not nullptr, the method disregards the IS_LENGTH_LIMITED flag and either stops searching at the location specified via ppEnd, or it updates ppEnd to provide the limit of any first matching content within this.  The method always diregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

Discussion

---

::casepFindSlice() methods for Uniseries content

This ::casepFindSlice() case-insensitive method returns a pointer to any first match on sPattern as a whole or between the locations indicated by the optional pBegin and ppEnd parameters.  If those parameters are nullptr, returns a pointer to any first match within this.  Returns nullptr in case the slice is not found.  The method disregards the IS_LENGTH_LIMITED flag, unless the ppEnd parameter is nullptr.  The method also disregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

---

::pFindWildSlice() methods for raw buffer content

Each of the ::pFindWildSlice() methods for targeted wildcard search scans a slice of the this content for a first match on the inbound search pattern slice, which is expected to contain at least one ‘*’ wildcard.  The search pattern slice also may include '?' wildcards.  If the optional *ppBegin and *ppEnd parameters are provided, the method disregards the IS_LENGTH_LIMITED flag and scans the slice designated by those parameters.  Otherwise the method scans the this content from its beginning.  The method returns a pointer to the location within this corresponding to any first matching ‘*’ wildcard.  If no match is found, or if no first ‘*’ wildcard is found in the search pattern slice, the method sets *ppBegin, *ppEnd, and *ppTarget to nullptr and returns nullptr.

Signature

Parameters

Discussion

---

::pFindWildSlice() methods for Uniseries content

This ::pFindWildSlice() method for targeted wildcard search returns a pointer to any first match on sPattern between the locations indicated by the optional ppBegin and ppEnd parameters.  If those parameters are nullptr, returns a pointer to any first match on sPattern within this.  Returns nullptr in case the slice is not found.  The method ignores the IS_LENGTH_LIMITED flag, unless the ppEnd parameter is nullptr.

Signature

Parameters

---

::casepFindWildSlice() methods for raw buffer content

Case-insensitive implementations of the ::pFindWildSlice() method for targeted wildcard search.  Each of the ::casepFindWildSlice() methods scans a slice of the this content for a first match on the inbound search pattern slice, which is expected to contain at least one ‘*’ wildcard.  The search pattern slice also may include '?' wildcards.  If the optional *ppBegin and *ppEnd parameters are provided, the method disregards the IS_LENGTH_LIMITED flag and scans the slice designated by those parameters.  Otherwise the method scans the this content from its beginning.  The method returns a pointer to the location within this corresponding to any first matching ‘*’ wildcard.  If no match is found, or if no first ‘*’ wildcard is found in the search pattern slice, the method sets *ppBegin, *ppEnd, and *ppTarget to nullptr and returns nullptr.  The method always diregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

Discussion

---

::casepFindWildSlice() methods for Uniseries content

This ::casepFindWildSlice() method performs a case-insensitive targeted wildcard search and returns a pointer to any first match on sPattern between the locations indicated by the optional ppBegin and ppEnd parameters.  If those parameters are nullptr, returns a pointer to any first match on sPattern within this.  Returns nullptr in case the slice is not found.  The method diregards the IS_LENGTH_LIMITED flag, unless the ppEnd parameter is nullptr.  The method always diregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

---

::pVectorizeWild() methods for raw buffer content

Each ::pVectorizeWild() method performs a vectorized wildcard search that returns a pointer to the first occurrence of the inbound search pattern slice, within a slice of the this content, and populates a MatchVector with beginning and ending locations of each match on each ‘*’ wildcard in the search pattern slice.  The method returns nullptr in case no such slice is found.  The search pattern slice comprises the bytes between the locations indicated by pPatternBegin and pPatternEnd.  The optional ppBegin parameter refers to a location, in the this content, to begin seeking a match.  If the optional ppEnd parameter is not nullptr, the method disregards the IS_LENGTH_LIMITED flag and either stops searching at the location specified via ppEnd, or it updates ppEnd to provide the limit of any first matching content within this.

Signature

Parameters

Discussion

---

::pVectorizeWild() method for Uniseries content

Performs a vectorized wildcard search by scaning the this content for a first match on the inbound search pattern passed in as a Uniseries instance.

Signature

Parameters

---

::casepVectorizeWild() methods for raw buffer content

Each ::casepVectorizeWild() method performs a case-insensitive vectorized wildcard search that returns a pointer to the first occurrence of the inbound search pattern slice, within a prospectively larger slice of the this content, and populates a MatchVector with beginning and ending locations of each match on each ‘*’ wildcard in the search pattern slice.  The method returns nullptr in case no such slice is found.  The search pattern slice comprises the bytes between the locations indicated by pPatternBegin and pPatternEnd.  The optional ppBegin parameter refers to a location, in the this content, to begin seeking a match.  If the optional ppEnd parameter is not nullptr, the method disregards the IS_LENGTH_LIMITED flag and either stops searching at the location specified via ppEnd, or it updates ppEnd to provide the limit of any first matching content within this.  The method always disregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

Discussion

---

::casepVectorizeWild() method for Uniseries content

Scans the this content for a first match on the inbound search pattern, which is expected to contain wildcards, and returns a pointer to the location within the the content corresponding to any first matching ‘*’ wildcard.  If no match is found, or if no first ‘*’ wildcard is found, this method sets *ppBegin, *ppEnd, and *ppTarget to nullptr and returns nullptr.

Signature

Parameters

---

::pSelectWild() methods for raw buffer content

Each of these ::pSelectWild() method searches hierarchical semistructured data, such as a loaded JSON or XML document, first for nested entries corresponding to leading and trailing ‘*’ wildcards within the search pattern, and then – once a caller-specified nesting level is reached – for a set of targets that match any remaining ‘*’ wildcards in the middle of the search pattern, which also can contain ‘?’ wildcards.  For each overall match on that inner portion of the search pattern, up to a defined MAX_MATCHES value, the method adds a MatchVector instance to pResultSet, which is a std::set provided by the caller.  If the hierarchy is completely traversed, the method returns a pointer to its beginning, setting ppEnd (if provided) to the location immediately after it and also passing back the populated result set via the pResultSet parameter.  Otherwise, the method sets pResultSet to nullptr and returns nullptr.

Signature

Parameters

Discussion

---

::pSelectWild() method for Uniseries content

This ::pSelectWild() method searches hierarchical semistructured data, such as a loaded JSON or XML document, first for nested entries corresponding to leading and trailing ‘*’ wildcards within the search pattern, and then – once a caller-specified nesting level is reached – for a set of targets that match any remaining ‘*’ wildcards in the middle of the search pattern, which also can contain ‘?’ wildcards.

Signature

Parameters

---

::casepSelectWild() methods for raw buffer content

Each of these ::casepSelectWild() methods performs a case-insensitive search through hierarchical semistructured data, such as a loaded JSON or XML document, first for nested entries corresponding to leading and trailing ‘*’ wildcards within the search pattern, and then – once a caller-specified nesting level is reached – for a set of targets that match any remaining ‘*’ wildcards in the middle of the search pattern, which also can contain ‘?’ wildcards.  Disregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

Discussion

---

::casepSelectWild() method for Uniseries content

This ::casepSelectWild() method performs a case-insensitive search through hierarchical semistructured data, such as a loaded JSON or XML document, first for nested entries corresponding to leading and trailing ‘*’ wildcards within the search pattern, and then – once a caller-specified nesting level is reached – for a set of targets that match any remaining ‘*’ wildcards in the middle of the search pattern, which also can contain ‘?’ wildcards.  Disregards the IS_CASE_SENSITIVE flag.

Signature

Parameters

---

Uniseries subscript operators

The non-const and const subscript (index) operators call IndexUtf8(), unless the entire content is 7-bit ASCII text.

Signature

Parameter

Discussion

---

::trim() method

The ::trim() method removes outboard white space from the this object’s content by replacing the code points with nulls.

Signature

Discussion

---

Uniseries validation and conversion methods

::validate() methods

Either of the ::validate() methods returns true if all of the this object’s content is valid UTF-8.

Signature

Parameter

Discussion

---

::convert8BitAscii() methods

In case content validation fails, the ::convert8BitAscii() method may serve as a reasonable fallback.  It calls one or another of the *Convert8BitAsciiToUtf8() functions.  It returns valid UTF-8 content and sets the optional iCount parameter reflecting the content’s length, as a count of its code points.

Signature

Parameter

---

Uniseries output stream operator

The output stream operator<< can be used to send the content of a Uniseries object to the console.

Signature

Discussion

---

Extending FastUtf8

In case you’ve got particular ASCII functionality in mind that you’d like to see UTF-8-enabled, and no Uniseries methods described here are quite like it, what might you do?  The aspects of ASCII character processing reimplemented to underlie the UTF-8 functionality described throughout this guide have included nothing more than char * pointer-based increment, compare, increment-and-compare, and decrement operations.  Where these operations occur in ASCII-specific code, a UTF-8-enabled version invokes these functions:

• CodePointAdvanceUtf8()
• CodePointCompareUtf8()
• CodePointAdvanceAndCompareUtf8()
• CodePointBacktrackUtf8()

These functions are defined near the top of fastutf8.cpp.  They’re slower than their char * counterparts, and on top of that – unlike the situation with ASCII – backtracking through UTF-8 content is considerably slower than advancing.  If you’d like your code to keep calling your ASCII-specific routine for 7-bit ASCII text and to call a UTF-8-ready routine just for multi-byte UTF-8 content, then your code can construct a Uniseries object, or an object of a Uniseries-based class, and can make use of that object’s IS_7BIT_CHAR_STRING flag.  The flag can guide the choice of which routine – UTF-8-enabled or ASCII-specific – to programmatically invoke.

The Uniseries iterator invokes the pointer-driven functions listed above and serves as their plainest interface at present.  If a need arises to derive a Uniseries-based class that builds directly on that pointer-driven functionality, an easy change to the class might involve wrapping those functions in Uniseries methods declared as protected.  A method in a derived class could, for now, use the Uniseries iterator to access individual code points within multi-byte content and could use char * pointers or the std::string iterator to access characters within 7-bit ASCII content.

Most code that makes the switch from ASCII to UTF-8 will encounter a slowdown, but with a Uniseries object – thanks to the flag-based mechanism I’ve just described – you can limit the slowdown almost entirely to the object’s construction phase.  That step of Uniseries construction involves UTF-8 validation.  But once that’s done, any remaining ASCII processing can happen with virtually no slowdown, even if your project includes code that’ll do revalidation for safety.  The slowdown of up-front UTF-8 validation is most prominent for short content snippets that get processed only once.  For larger content that comes to reside in reusable Uniseries objects, your UTF-8-ready code can run with about the same 7-bit ASCII performance as it did prior to UTF-8-enablement.

The complete FastUtf8 source code is available at GitHub > kirkjkrauss > FastUtf8.  The above source code listings are extracted from the included code in Demos. The listings are formatted using the SyntaxHighlighter library, copyright (c) 2004-2013, Alex Gorbatchev.

All other materials copyright © 2026 developforperformance.com.

C++ and its logo are trademarks of the Standard C++ Foundation.  Windows® and Visual Studio® are trademarks or registered trademarks of Microsoft Corp.  Unix® is a registered trademark of The Open Group.  Linux® is a registered trademark of Linus Torvalds.  Ubuntu® is a registered trademark of Canonical Ltd.

Develop for Performance