GeezerVison is showing github.com — [SQL] Compute using UTF-8 strings; support `STARTSWITH` and `ENDSWITH` by mihaibudiu · Pull Request #7729 · feldera/feldera
[SQL] Compute using UTF-8 strings; support STARTSWITH and ENDSWITH
#7729
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement . We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Uh oh!
There was an error while loading. Please reload this page.
Show file tree
Hide file tree
Changes from all commits
Commits
File filter
Filter by extension
.java
(22)
.md
(3)
.rs
(2)
All 3 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Hide whitespace
Diff view
Hide whitespace
There are no files selected for viewing
Original file line number
Diff line number
Diff line change
Expand Up
@@ -509,6 +509,20 @@ pub fn binary_position__(needle: ByteArray, haystack: ByteArray) -> i32 {
some_function2!(binary_position, ByteArray, ByteArray, i32);
#[doc(hidden)]
pub fn binary_starts_with__(value: ByteArray, prefix: ByteArray) -> bool {
value.as_slice().starts_with(prefix.as_slice())
}
some_function2!(binary_starts_with, ByteArray, ByteArray, bool);
#[doc(hidden)]
pub fn binary_ends_with__(value: ByteArray, suffix: ByteArray) -> bool {
value.as_slice().ends_with(suffix.as_slice())
}
some_function2!(binary_ends_with, ByteArray, ByteArray, bool);
#[doc(hidden)]
pub fn binary_substring2__(source: ByteArray, left: i32) -> ByteArray {
// SQL indexing starts at 1
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -383,6 +383,20 @@ pub fn position__(needle: SqlString, haystack: SqlString) -> i32 {
some_function2!(position, SqlString, SqlString, i32);
#[doc(hidden)]
pub fn starts_with__(value: SqlString, prefix: SqlString) -> bool {
value.str().starts_with(prefix.str())
}
some_function2!(starts_with, SqlString, SqlString, bool);
#[doc(hidden)]
pub fn ends_with__(value: SqlString, suffix: SqlString) -> bool {
value.str().ends_with(suffix.str())
}
some_function2!(ends_with, SqlString, SqlString, bool);
#[doc(hidden)]
pub fn char_length_(value: SqlString) -> i32 {
value.str().chars().count() as i32
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -51,6 +51,11 @@ aggregation functions `BIT_AND`, `BIT_OR`, and `BIT_XOR`.
<td>Concatenation of binary values</td>
<td><code>x'ab' || x'cd'</code> => <code>x'ABCD'</code></td>
</tr>
<tr>
<td><a id="endswith"></a><code>ENDSWITH</code>(binary, suffix)</td>
<td>Returns <code>TRUE</code> if the bytes of <code>binary</code> end with the bytes of <code>suffix</code>. An empty <code>suffix</code> gives <code>TRUE</code>. If any argument is <code>NULL</code>, returns <code>NULL</code>.</td>
<td><code>ENDSWITH(x'abcdef', x'ef')</code> => <code>TRUE</code></td>
</tr>
<tr>
<td><a id="gunzip"></a><code>GUNZIP</code>(binary)</td>
<td>
Expand Down
Expand Up
@@ -99,6 +104,11 @@ aggregation functions `BIT_AND`, `BIT_OR`, and `BIT_XOR`.
<td>Returns last <code>count</code> bytes in the byte array. If any argument is <code>NULL</code>, return <code>NULL</code>.</td>
<td><code>right(x'abcdef', 2)</code> => <code>cdef</code></td>
</tr>
<tr>
<td><a id="startswith"></a><code>STARTSWITH</code>(binary, prefix)</td>
<td>Returns <code>TRUE</code> if the bytes of <code>binary</code> start with the bytes of <code>prefix</code>. An empty <code>prefix</code> gives <code>TRUE</code>. If any argument is <code>NULL</code>, returns <code>NULL</code>.</td>
<td><code>STARTSWITH(x'abcdef', x'ab')</code> => <code>TRUE</code></td>
</tr>
<tr>
<td><a id="substring"></a><code>SUBSTRING</code>(binary FROM integer)</td>
<td>Generate a substring of binary starting at the given offset in bytes. The first offset is 1. If the start position integer is less than 1, it is treated as 1</td>
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -104,6 +104,7 @@
* `DOW`: [datetime](datetime.md#date_dayofweek), [datetime](datetime.md#timestamp_dayofweek)
* `DOY`: [datetime](datetime.md#doy)
* `ELEMENT`: [array](array.md#element)
* `ENDSWITH`: [string](string.md#endswith), [binary](binary.md#endswith)
* `EQUALS`: [datetime](datetime.md#equals)
* `ESCAPE`: [string](string.md#like)
* `EVERY` (aggregate): [aggregates](aggregates.md#every)
Expand Down
Expand Up
@@ -247,6 +248,7 @@
* `SPLIT`: [string](string.md#split)
* `SPLIT_PART`: [string](string.md#split_part)
* `SQRT`: [float](float.md#sqrt)
* `STARTSWITH`: [string](string.md#startswith), [binary](binary.md#startswith)
* `STDDEV` (aggregate): [aggregates](aggregates.md#stddev)
* `STDDEV_POP` (aggregate): [aggregates](aggregates.md#stddev_pop)
* `STDDEV_SAMP` (aggregate): [aggregates](aggregates.md#stddev_samp)
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -61,7 +61,9 @@ A string literal prefixed with the `U&` can contain Unicode characters
described using their numeric code: `U&'hello\0041'` includes the
Unicode character with code U+0041, which is the uppercase letter 'A'.
A Unicode character is described by an escape character followed by
four hexadecimal digits.
four hexadecimal digits. A character above U+FFFF is described by two
escapes, the UTF-16 surrogate pair of the character: `U&'\D83D\DC0D'`
is the character U+1F40D, which is 🐍.
The default escape character is backslash, but it can be changed using
the following syntax: `U&'hello!0041' UESCAPE '!'`. The `UESCAPE`
Expand Down
Expand Up
@@ -142,6 +144,11 @@ example.
<td>String concatenation with separator <code>sep</code>. Can have any number of arguments. <code>sep</code> is intercalated between all strings. If <code>sep</code> is <code>NULL</code> result is <code>NULL</code>. Other <code>NULL</code> arguments are ignored.</td>
<td><code>CONCAT_WS(',', 'Post', 'greSQL', NULL, '1')</code> => <code>Post,greSQL,1</code></td>
</tr>
<tr>
<td><a id="endswith"></a><code>ENDSWITH(string, suffix)</code></td>
<td>Returns <code>TRUE</code> if <code>string</code> ends with <code>suffix</code>. The comparison is exact: it is case-sensitive and it does not normalize Unicode characters. An empty <code>suffix</code> gives <code>TRUE</code>. If any argument is <code>NULL</code>, returns <code>NULL</code>.</td>
<td><code>ENDSWITH('Feldera', 'era')</code> => <code>TRUE</code></td>
</tr>
<tr>
<td><a id="initcap"></a><code>INITCAP ( string )</code></td>
<td>Converts the first letter of each word in <code>string</code> to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.</td>
Expand Down
Expand Up
@@ -241,6 +248,11 @@ example.
<code>SPLIT_PART('a|b|c|', '|', 5)</code> => <code>''</code>
</td>
</tr>
<tr>
<td><a id="startswith"></a><code>STARTSWITH(string, prefix)</code></td>
<td>Returns <code>TRUE</code> if <code>string</code> starts with <code>prefix</code>. The comparison is exact: it is case-sensitive and it does not normalize Unicode characters. An empty <code>prefix</code> gives <code>TRUE</code>. If any argument is <code>NULL</code>, returns <code>NULL</code>.</td>
<td><code>STARTSWITH('Feldera', 'Fel')</code> => <code>TRUE</code></td>
</tr>
<tr>
<td><a id="substr"></a><code>SUBSTR (</code> string, start <code> [ ,</code> length <code>] )</code></td>
<td>Extracts the substring of <code>string</code> starting at the <code>start</code>'th character, and stopping after <code>length</code> characters if the value is specified. If <code>start</code> is negative, the first character is chosen counting backwards from the end of <code>string</code>. If <code>length</code> is negative the empty string is returned. The index of the first character is 1.</td>
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -19,6 +19,7 @@
import org.dbsp.sqlCompiler.ir.type.primitive.DBSPTypeVariant;
import org.dbsp.util.IJson;
import org.dbsp.util.Linq;
import org.dbsp.util.Utf8String;
import org.dbsp.util.Utilities;
import javax.annotation.Nullable;
Expand Down
Expand Up
@@ -134,8 +135,9 @@ public boolean isFalsy(String variable) {
} else if (expression.is(DBSPStringLiteral.class)) {
// Quoted SET values: 'off', 'false', and '0' disable a feature,
// matching the unquoted keywords.
String value = expression.to(DBSPStringLiteral.class).value;
if (value == null) return true;
Utf8String text = expression.to(DBSPStringLiteral.class).value;
if (text == null) return true;
String value = text.toJavaString();
return value.equalsIgnoreCase("off") || value.equalsIgnoreCase("false")
|| value.equals("0") || value.isEmpty();
} else {
Expand All
@@ -155,14 +157,8 @@ static Integer asInt(DBSPExpression expression) {
}
}
if (expression.is(DBSPStringLiteral.class)) {
String value = expression.to(DBSPStringLiteral.class).value;
if (value == null)
return null;
try {
return Integer.parseInt(value.trim());
} catch (NumberFormatException ignored) {
return null;
}
Utf8String value = expression.to(DBSPStringLiteral.class).value;
return value == null ? null : value.asInt();
}
return null;
}
Expand Down
Expand Up
@@ -231,10 +227,10 @@ public List<String> disabledOptimizations() {
DBSPExpression expression = Utilities.getExists(this.variables, canonicalVariableName(DISABLE_OPTIMIZATIONS));
if (!expression.is(DBSPStringLiteral.class))
return List.of();
String value = expression.to(DBSPStringLiteral.class).value;
Utf8String value = expression.to(DBSPStringLiteral.class).value;
if (value == null)
return List.of();
List<String> names = Linq.map(Arrays.asList(value.split(",")), String::trim);
List<String> names = Linq.map(Arrays.asList(value.toJavaString(). split(",")), String::trim);
Comment thread
mihaibudiu marked this conversation as resolved.
Resolved
Uh oh!
There was an error while loading. Please reload this page.
return Linq.where(names, name -> !name.isEmpty());
}
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -129,7 +129,7 @@ public VisitDecision preorder(DBSPDoubleLiteral literal) {
@Override
public VisitDecision preorder(DBSPStringLiteral literal) {
if (literal.value != null)
this.appendable.append(Utilities.doubleQuote(literal.value, false));
this.appendable.append(Utilities.doubleQuote(literal.value.toJavaString() , false));
else
this.appendable.append(this.nullRepresentation.get());
return VisitDecision.STOP;
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -404,10 +404,10 @@ public void postorder(DBSPSimplePathSegment node) {
public void postorder(DBSPStringLiteral node) {
if (node.value != null) {
this.property("value");
this.stream.append(node.value);
this.stream.append(node.value.toJavaString() );
}
this.property("charset");
this.stream.append(node.charset.toString() );
this.stream.append("UTF-8" );
super.postorder(node);
}
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -114,7 +114,7 @@ public VisitDecision preorder(DBSPDoubleLiteral literal) {
@Override
public VisitDecision preorder(DBSPStringLiteral literal) {
if (literal.value != null)
this.appendable.append("'" + literal.value.replace("'", "''") + "'");
this.appendable.append("'" + literal.value.toJavaString(). replace("'", "''") + "'");
else
this.appendable.append(DBSPNullLiteral.NULL);
return VisitDecision.STOP;
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -1302,12 +1302,9 @@ public VisitDecision preorder(DBSPStringLiteral literal) {
if (literal.isNull())
return this.doNull(literal);
this.push(literal);
Objects.requireNonNull(literal.value);
byte[] bytes = literal.value.getBytes(literal.charset);
String decoded = new String(bytes, literal.charset);
decoded = Utilities.doubleQuote(decoded, true);
String quoted = Utilities.doubleQuote(Objects.requireNonNull(literal.value).toJavaString(), true);
this.builder.append(literal.wrapSome(
DBSPTypeCode.STRING.rustName + "::from(arcstr::literal!(" + decoded + "))"));
DBSPTypeCode.STRING.rustName + "::from(arcstr::literal!(" + quoted + "))"));
this.pop(literal);
return VisitDecision.STOP;
}
Expand Down
Original file line number
Diff line number
Diff line change
Expand Up
@@ -53,6 +53,7 @@
import org.dbsp.sqlCompiler.compiler.errors.UnimplementedException;
import org.dbsp.sqlCompiler.compiler.errors.UnsupportedException;
import org.dbsp.sqlCompiler.compiler.frontend.calciteCompiler.functions.ExternalFunction;
import org.dbsp.sqlCompiler.compiler.frontend.calciteCompiler.NonBmpLiterals;
import org.dbsp.sqlCompiler.compiler.frontend.calciteCompiler.WarnFloatingPointEquality;
import org.dbsp.sqlCompiler.compiler.frontend.calciteCompiler.ProgramIdentifier;
import org.dbsp.sqlCompiler.compiler.frontend.calciteObject.CalciteObject;
Expand Down
Expand Up
@@ -136,13 +137,13 @@
import org.dbsp.util.IWritesLogs;
import org.dbsp.util.Linq;
import org.dbsp.util.Logger;
import org.dbsp.util.Utf8String;
import org.dbsp.util.Utilities;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.Point;
import javax.annotation.Nullable;
import java.math.BigDecimal;
import java.nio.charset.Charset;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
Expand Down
Expand Up
@@ -317,10 +318,8 @@ public DBSPExpression visitLiteral(RexLiteral literal) {
case REAL:
return new DBSPRealLiteral(Objects.requireNonNull(literal.getValueAs(Float.class)));
case STRING:
String str = literal.getValueAs(String.class);
RelDataType litType = literal.getType();
Charset charset = litType.getCharset();
return new DBSPStringLiteral(Objects.requireNonNull(str), Objects.requireNonNull(charset), type);
String str = Objects.requireNonNull(literal.getValueAs(String.class));
return new DBSPStringLiteral(CalciteObject.EMPTY, type, Utf8String.fromJava(str));
case BOOL:
return new DBSPBoolLiteral(Objects.requireNonNull(literal.getValueAs(Boolean.class)));
case DECIMAL:
Expand Down
Expand Up
@@ -785,13 +784,19 @@ static DBSPExpression compileFunction(
* @param expectedArgCount A list containing all known possible argument counts.
*/
static DBSPExpression compileStrictFunction(
boolean adjustNullability, RexCall call , CalciteObject node, DBSPType resultType,
boolean adjustNullability, String baseName , CalciteObject node, DBSPType resultType,
List<DBSPExpression> ops, Integer... expectedArgCount) {
for (DBSPExpression op: ops) {
if (op.is(DBSPLiteral.class) && op.to(DBSPLiteral.class).isNull())
return resultType.none();
}
return compileFunction(adjustNullability, call, node, resultType, ops, expectedArgCount);
return compileFunction(adjustNullability, baseName, node, resultType, ops, expectedArgCount);
}
static DBSPExpression compileStrictFunction(
boolean adjustNullability, RexCall call, CalciteObject node, DBSPType resultType,
List<DBSPExpression> ops, Integer... expectedArgCount) {
return compileStrictFunction(adjustNullability, getCallName(call), node, resultType, ops, expectedArgCount);
}
static DBSPExpression compileStrictFunction(
Expand Down
Expand Up
@@ -864,7 +869,8 @@ void ensureKeyword(CalciteObject node, List<DBSPExpression> ops, int argument) {
if (arg.getType().sameType(DBSPTypeKeyword.INSTANCE))
return;
if (arg.is(DBSPStringLiteral.class)) {
ops.set(argument, new DBSPKeywordLiteral(node, Objects.requireNonNull(arg.to(DBSPStringLiteral.class).value)));
Utf8String keyword = Objects.requireNonNull(arg.to(DBSPStringLiteral.class).value);
ops.set(argument, new DBSPKeywordLiteral(node, keyword.toJavaString()));
return;
}
throw new CompilationError("Unknown keyword", node);
Expand Down
Expand Up
@@ -944,9 +950,10 @@ static CompilationError operandCountError(CalciteObject node, String name, int o
void checkRegexLiteral(CalciteObject call, DBSPExpression arg) {
if (!arg.is(DBSPStringLiteral.class))
return;
String pattern = arg.to(DBSPStringLiteral.class).value;
if (pattern == null)
Utf8String value = arg.to(DBSPStringLiteral.class).value;
if (value == null)
return;
String pattern = value.toJavaString();
String construct = RustRegexChecker.unsupportedConstruct(pattern);
if (construct != null)
this.compiler.reportWarning(call.getPositionRange(), RustRegexChecker.WARNING,
Expand Down
Expand Up
@@ -1066,14 +1073,14 @@ void checkFormatArg(CalciteObject call, List<DBSPExpression> args, int formatPos
return;
}
if (formatArg.is(DBSPStringLiteral.class)) {
String str = formatArg.to(DBSPStringLiteral.class).value;
Utf8String str = formatArg.to(DBSPStringLiteral.class).value;
if (str == null) {
this.compiler.reportWarning(formatArgPosition,
"Suspicious argument",
"Format argument is NULL.");
return;
}
if (!str.contains("%")) {
if (!str.toJavaString(). contains("%")) {
this.compiler.reportWarning(formatArgPosition,
"Suspicious argument",
"Format argument does not look like a format string.");
Expand All
@@ -1092,8 +1099,8 @@ void checkFormatArg(CalciteObject call, List<DBSPExpression> args, int formatPos
if (args.size() == 2) {
DBSPExpression otherArg = args.get(1 - formatPosition);
if (otherArg.is(DBSPStringLiteral.class)) {
String strData = otherArg.to(DBSPStringLiteral.class).value;
if (strData != null && strData.contains("%")) {
Utf8String strData = otherArg.to(DBSPStringLiteral.class).value;
if (strData != null && strData.toJavaString(). contains("%")) {
this.compiler.reportWarning(argumentPosition(call, otherArg),
"Suspicious argument",
"Are the two arguments swapped?");
Expand All
@@ -1117,6 +1124,10 @@ public DBSPExpression visitCall(RexCall call) {
.append(" ")
.appendSupplier(() -> finalCall.getType().toString());
List<DBSPExpression> ops = Linq.map(call.operands, e -> e.accept(this));
if (call.op == NonBmpLiterals.Wrapper.INSTANCE) {
DBSPStringLiteral literal = ops.get(0).to(DBSPStringLiteral.class);
return new DBSPStringLiteral(node, type, literal.value);
}
String operationName = call.op.kind.sql;
switch (call.op.kind) {
case CHECKED_TIMES:
Expand Down
Expand Up
@@ -1969,15 +1980,19 @@ else if (arg0Type.is(DBSPTypeMap.class))
ops.set(1, this.makeRegex(node, ops.get(1)));
return compileFunction(call, node, type, ops, 2);
}
case POSITION: {
case POSITION:
case STARTS_WITH:
case ENDS_WITH: {
// Also called for STARTSWITH and ENDSWITH
validateArgCount(node, operationName, ops.size(), 2);
String module_prefix = "";
if (ops.get(0) .type.is(DBSPTypeBinary.class)) {
if (Linq.any(ops, op -> op .type.is(DBSPTypeBinary.class) )) {
module_prefix = "binary_";
} else {
this.ensureString(ops, 0);
this.ensureString(ops, 1);
}
return compileFunction( module_prefix + getCallName(call), node, type, ops, 2);
return compileStrictFunction(true, module_prefix + getCallName(call), node, type, ops, 2);
}
case ARRAY_JOIN:
case ARRAY_TO_STRING: {
Expand Down
Expand Up
@@ -2086,7 +2101,7 @@ else if (arg0Type.is(DBSPTypeMap.class))
Utilities.enforce(index.is(DBSPStringLiteral.class));
DBSPStringLiteral lit = index.to(DBSPStringLiteral.class);
Utilities.enforce(lit.value != null);
String fieldName = lit.value;
String fieldName = lit.value.toJavaString() ;
Utilities.enforce(tuple.originalStruct != null);
DBSPTypeStruct.Field field = tuple.originalStruct.getField(
new ProgramIdentifier(fieldName));
Expand Down
Loading
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Loading
Oops, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.