Writing from ADLS CSV to Synapse Dedicated SQL Pool – Unexpected error encountered filling record reader buffer: HadoopSqlException: Arithmetic overflow error converting expression to data type NVARCHAR
We are copying over email records from Dataverse (D365) using Synapse Link for Dataverse which copies incremental changes to ADLS and then to a Dedicated SQL pool in Synapse using a data flow.
The email body can be incredibly long, and our table in the sql pool accounts for this by using a field type of nvarchar(max). However, when copying this data over using a Data Flow in Synapse, we encounter the following error
Unexpected error encountered filling record reader buffer: HadoopSqlException: Arithmetic overflow error converting expression to data type NVARCHAR
If we truncate the email body to 10,000 characters, the copy performs perfectly fine.
Is there a way to copy the entire email body over without having to restrict the length? I understand that Polybase previously had a limitation of 1MB and Hadoop still has that limitation. Is this something that we are stuck with or do we need to research other methods of copying over this data?
Read more here: Source link