ArchitectureA supervised JVM-class runtime — OLTP on seven engines, OLAP on three. AI-native, MCP-native, observable as plain SQL.Read the architecture
Está viendo la edición Perú. Está viendo la edición Colombia. You're viewing the Pakistan edition. Cambiar a la edición global →Cambiar a la edición global →Switch to the global edition →
Data and enginesProduct Release2 min read

Result sets export to Parquet and Parquet files import, without Hadoop

A result set writes to Parquet and a Parquet file loads back, through one small library with no Hadoop. Types, sizes and nullability travel in the footer, and SNAPPY files from pandas, Spark and DuckDB import as shipped.

Analytics teams exchange data as Parquet: pandas, Spark and DuckDB read and write it natively. Until now the platform's result-set readers and writers spoke CSV, Excel, JSON and XML. This release adds Parquet in both directions, through a single library with no Hadoop dependency, so a script exports a query to the lakehouse and loads a file back without a cluster on the classpath.

Export

  • Every JDBC type mapped Each column maps to a Parquet type and is written optional. The source type, size, scale and nullability are stored in the file footer, so an import restores CHAR(10), NUMERIC and NOT NULL as they were.
  • SNAPPY by default SNAPPY is the export default and its codec ships with the platform, so the files pandas, Spark and DuckDB write import as shipped. GZIP files are read too.
  • No partial files A failed export discards the target rather than closing it, because closing writes a footer and would publish the rows written so far as a valid file.
  • Engine detail handled Informix floating decimals, reported at scale 8 by the metadata layer, are rounded half-up to it. Unsigned integers are recognised from the driver's type name.

Import

  • Typed from the file Columns take their types from the file schema, and from the platform's own footer metadata when it is present.
  • Nested columns refused unless excluded, so a file carrying a struct or a list fails clearly instead of loading a column of blobs.
  • A missing codec named A file compressed with a codec the server lacks surfaces as an error naming the jar to add.
  • Verified independently Every file the test suite writes is checked by a reader that parses the footer apart from the import. Fixtures written by pyarrow 21 in GZIP and SNAPPY round-trip, and five Informix suites cover the round trip, query export, foreign-file import, table restore and volume.

Upsert keeps what it should

  • Columns excluded from update An upsert now accepts the columns that keep their stored value when the row already exists, a creation timestamp or the user who created the row, while still inserting them for a new row. Until now only the serial column was ever excluded.
  • Existing callers unchanged The serial column stays excluded whether or not it is named, and the previous form passes no extra exclusions.

Scripts reach the Parquet reader and writer the same way they reach CSV and Excel, and the script reference now describes every reader format.

On the platformDeveloper ExperiencePlatform

All posts