PySpark usage of like, ilike, rlike and not like
Using a sample pyspark Dataframe
ILIKE
(from 3.3.0)
SQL ILIKE expression (case insensitive LIKE). Returns a boolean Column based on a case insensitive match.
df1.filter(df1.firstname.ilike('%Ria')).collect()
RLIKE
We can get similar match with RLIKE
df1.filter(df1.firstname.rlike('(?i)Ria$')).collect()
LIKE
Case sensitive match
df1.filter(df1.firstname.like('%Ria')).collect()
with like as expression
df = df1.filter("firstname like '%Ria%'")
df.collect()
Not Like
There is nothing like notlike function, however negation of Like can be used to achieve this, using the '~'operator
df1.filter(~ df1.firstname.like('%Ria')).collect()
SQL we can use *not like *
df = df1.filter("firstname not like '%Ria%'"
df.collect())
Notebook
AI Engineer & Quantum Ambassador at IBM
2yThis was useful, thank you!
Azure Data Engineer@TechM | Ex HCL | Ex Cognizant | PySpark | HDFS | Hive | Spark | Python | Sqoop | Azure Cloud | SQL | Hadoop Ecosystem
2yInformative