google cloud platform – Data Fusion UTF-8 vs UTF-16 encoded file processing issue

In my Data Fusion pipeline,
I start by reading a GCS file as a blob.
Since the file is encoded in UTF-16, I set the charset to UTF-16 during the transformation step. Wrangler does not indicate any issues.

For testing purposes, I created a file with various formatting errors and ran my pipeline against it. Instead of processing each issued record separately as expected, the entire input is outputted as a single row. No correct rows are being written (in my case to BigQuery).

When I manually changed the file’s charset in Sublime Text to UTF-8 and read GCS file as text, the pipeline behaves as expected, correctly treating each row as a separate record in the error output and also writing correct rows to its destination.

I even tried adding another transformation step that would set charset and output bytes again but that every times results in the same – no correct records on the output, all errors are treated as 1.

Does this ring any bell with anyone?