You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I put the file in a partitioned directory, tst-data/date=2024-01-01/file.parquet, opened it in Tad, the UI showed the date column data type was Date.
But when I moved the same file to a non-partitioned directory, tst-data/file.parquet, Tad showed the date column data type was varchar.
This was misleading. When I viewed a partitioned parquet file generated by my Apache Spark job, I thought my Spark job output incorrect data type. And it took me some time to figure out it was Tad.
Expected Behavior
As a file viewer, TAD should only display data types as they are, and should not infer data types from partitioned directories
The text was updated successfully, but these errors were encountered:
Background
I have a parquet file and here is the schema.
When I put the file in a partitioned directory,
tst-data/date=2024-01-01/file.parquet
, opened it in Tad, the UI showed the date column data type wasDate
.But when I moved the same file to a non-partitioned directory,
tst-data/file.parquet
, Tad showed the date column data type was varchar.This was misleading. When I viewed a partitioned parquet file generated by my Apache Spark job, I thought my Spark job output incorrect data type. And it took me some time to figure out it was Tad.
Expected Behavior
As a file viewer, TAD should only display data types as they are, and should not infer data types from partitioned directories
The text was updated successfully, but these errors were encountered: