I was trying to get the delta compression type during read and writes, the underlying file format is parquet, setting compression type globally using 'spark.sql.parquet.compression.codec'='zstd' works ...