python – How to solve this Pyspark Code Block using Regexp

I have this CSV file

enter image description here

but when I am running my notebook regex shows some error

from pyspark.sql.functions import regexp_replace

dff ="header", "true").option("inferSchema", "true").option('multiline', 'true').option('encoding', 'UTF-8').option("delimiter", "‡‡,‡‡").csv(path)
#dffs_headers = dff.dtypes

for i in dffs_headers:
  columnLabel = i[0]
  newColumnLabel = columnLabel.replace('‡‡','').replace('‡‡','')
  if columnLabel != newColumnLabel:
    dff = dff.drop(columnLabel)

As and a result I am getting this

enter image description here

Can anyone improvise this code, it will be a great help.

Expected output is

|��123456��,��Version2��,��All questions have been answered accurately and the guidance in the questionnaire was understood and followed��,��2010-12-16 00:01:48.020000000��|

But I am getting


Second column is showing Truncated value

Read more here: Source link