Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Snowflake JDBC Connector #10387

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
1d37327
Adding Bloomberg Snowflake connector.
dprophet Mar 17, 2023
3bf24da
Fix version number problem with build.
dprophet Mar 17, 2023
2afade4
Add Snowflake JDBC Connector
yuuteng Dec 22, 2021
e68f59c
Update trino snapshot version to 372
yuuteng Feb 18, 2022
bd91c9d
Update format of the doc of snowflake
yuuteng Feb 18, 2022
409acde
Update trino jdbc library import
yuuteng Feb 18, 2022
f08dd2a
Fix date formatter from yyyy to uuuu
yuuteng Feb 21, 2022
ee9b920
Fix date test case
yuuteng Feb 21, 2022
b1eecc2
Remove defunct property allow-drop-table
yuuteng Feb 21, 2022
b902d0d
Update trino version to 374
yuuteng Mar 14, 2022
e604633
Update snowflake config to adapt 374
yuuteng Mar 14, 2022
61ac1f0
Update the range of the test of Date type
yuuteng Mar 14, 2022
92d42a1
Update to version 375
yuuteng Mar 22, 2022
ad4bbc3
Fix snowflake after updating to 375
yuuteng Mar 22, 2022
37322dc
Update to 381
yuuteng May 16, 2022
68249ba
Fix mvn pom import
yuuteng May 16, 2022
7bab04d
Format snowflake.rst
yuuteng May 16, 2022
befa51f
Reorderd Data tests in type mapping
yuuteng May 16, 2022
c7cb13b
Update function code
yuuteng May 16, 2022
7968933
Add product test
yuuteng May 16, 2022
4814f37
Rename product test tablename
yuuteng May 16, 2022
f4d9f15
Add Env, Suite and properties of Snowflake for production test
yuuteng May 16, 2022
7d73f13
Add trinoCreateAndInsert()
yuuteng May 16, 2022
9495db8
Refactor snowflake from single node to multi node
yuuteng May 17, 2022
d61b8f6
Pass product tests
yuuteng May 18, 2022
c44be40
Removed snowflake.properties in trino server dev
yuuteng May 18, 2022
391f4c5
Resolved issues 19 05 2022 and fixed tests
yuuteng May 19, 2022
517ded4
Remove Types.VARBINARY
yuuteng May 20, 2022
99921d3
Add private static SliceWriteFunction charWriteFunction
yuuteng May 31, 2022
5b83fff
Update test case
yuuteng May 31, 2022
83bf784
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng Jun 16, 2022
68c7587
Update docs/src/main/sphinx/connector/snowflake.rst
yuuteng Jun 16, 2022
3d62bd5
Update plugin/trino-snowflake/pom.xml
yuuteng Jun 16, 2022
e9d2897
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng Jun 16, 2022
49e9325
Resolved review open issues
yuuteng Jun 16, 2022
14b654d
Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case
yuuteng Jun 16, 2022
f83dd03
Updated properties file
yuuteng Jun 17, 2022
8c3c815
Updated properties files
yuuteng Jun 17, 2022
fb9c4a2
Renamed properties in Testing
yuuteng Jun 17, 2022
e2ae8d4
Revert "Renamed properties in Testing"
yuuteng Jun 17, 2022
bddbb92
Renamed properties and fixed tests
yuuteng Jun 17, 2022
9937df0
Update the way to pass ENV values for production test
yuuteng Jun 24, 2022
edb1a79
Update trino version to 388
yuuteng Jun 24, 2022
b375123
Update Trino version 391
yuuteng Jul 18, 2022
95a943c
Update trino version to 394
yuuteng Aug 26, 2022
73c3152
Update to 395
yuuteng Aug 31, 2022
2a9802a
Update to 411
yuuteng Mar 22, 2023
9ddc0c4
Update and fix errors
yuuteng Mar 27, 2023
c5729e3
Build successfully with 411
yuuteng Mar 27, 2023
9002e50
Merge branch 'snowflake' of https://github.com/bloomberg/trino into b…
yuuteng Mar 27, 2023
632a646
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng Mar 27, 2023
901aa0b
Merge pull request #2 from yuuteng/feature/2551-add-snowflake-connector
dprophet Mar 27, 2023
1687bea
Adding varchar type to mapping Snowflake.
dprophet Mar 28, 2023
acf5182
Fixing some tests.
dprophet Apr 3, 2023
bc767c0
Fix TestSnowflakeConnectorTest.
yuuteng Apr 5, 2023
c071cd8
Fix type mapper
yuuteng Apr 6, 2023
770c467
Fix testconnector
yuuteng Apr 6, 2023
f3236a4
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng Apr 6, 2023
14e7431
Add Snowflake JDBC Connector
yuuteng Dec 22, 2021
7671dfe
Update trino snapshot version to 372
yuuteng Feb 18, 2022
951aa96
Update format of the doc of snowflake
yuuteng Feb 18, 2022
fbde6f0
Update trino jdbc library import
yuuteng Feb 18, 2022
d0d82d6
Fix date formatter from yyyy to uuuu
yuuteng Feb 21, 2022
6cb0ca2
Fix date test case
yuuteng Feb 21, 2022
187e04b
Remove defunct property allow-drop-table
yuuteng Feb 21, 2022
d1d6f5c
Update trino version to 374
yuuteng Mar 14, 2022
a10a085
Update snowflake config to adapt 374
yuuteng Mar 14, 2022
38e068f
Update the range of the test of Date type
yuuteng Mar 14, 2022
765dd86
Update to version 375
yuuteng Mar 22, 2022
410df33
Fix snowflake after updating to 375
yuuteng Mar 22, 2022
c82e5fd
Update to 381
yuuteng May 16, 2022
7498c32
Fix mvn pom import
yuuteng May 16, 2022
883219a
Format snowflake.rst
yuuteng May 16, 2022
cc9d2b2
Reorderd Data tests in type mapping
yuuteng May 16, 2022
3db806c
Update function code
yuuteng May 16, 2022
47b02ef
Add product test
yuuteng May 16, 2022
e35e282
Rename product test tablename
yuuteng May 16, 2022
eae4f64
Add Env, Suite and properties of Snowflake for production test
yuuteng May 16, 2022
1c4892f
Add trinoCreateAndInsert()
yuuteng May 16, 2022
f775981
Refactor snowflake from single node to multi node
yuuteng May 17, 2022
f107647
Pass product tests
yuuteng May 18, 2022
e4534bf
Removed snowflake.properties in trino server dev
yuuteng May 18, 2022
b670b22
Resolved issues 19 05 2022 and fixed tests
yuuteng May 19, 2022
502425c
Remove Types.VARBINARY
yuuteng May 20, 2022
e0f3137
Add private static SliceWriteFunction charWriteFunction
yuuteng May 31, 2022
2830740
Update test case
yuuteng May 31, 2022
a75b731
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng Jun 16, 2022
f2e1f9b
Update docs/src/main/sphinx/connector/snowflake.rst
yuuteng Jun 16, 2022
f6c6fee
Update plugin/trino-snowflake/pom.xml
yuuteng Jun 16, 2022
642d61f
Update plugin/trino-snowflake/src/main/java/io/trino/plugin/snowflake…
yuuteng Jun 16, 2022
b832673
Resolved review open issues
yuuteng Jun 16, 2022
c50c9ff
Disabled JDBC_TREAT_DECIMAL_AS_INT and fixed test case
yuuteng Jun 16, 2022
0b4a89f
Updated properties file
yuuteng Jun 17, 2022
c6f2c53
Updated properties files
yuuteng Jun 17, 2022
a61977b
Renamed properties in Testing
yuuteng Jun 17, 2022
515d67e
Revert "Renamed properties in Testing"
yuuteng Jun 17, 2022
cf9e4a4
Renamed properties and fixed tests
yuuteng Jun 17, 2022
c7fd30d
Update the way to pass ENV values for production test
yuuteng Jun 24, 2022
327cebf
Update trino version to 388
yuuteng Jun 24, 2022
b469b41
Update Trino version 391
yuuteng Jul 18, 2022
44b1ef8
Update trino version to 394
yuuteng Aug 26, 2022
ceadf76
Update to 395
yuuteng Aug 31, 2022
1b4c80a
Update to 411
yuuteng Mar 22, 2023
0327ab8
Update and fix errors
yuuteng Mar 27, 2023
baaac25
Build successfully with 411
yuuteng Mar 27, 2023
1a82298
Adding Bloomberg Snowflake connector.
dprophet Mar 17, 2023
11f220a
Adding varchar type to mapping Snowflake.
dprophet Mar 28, 2023
1d649b7
Fixing some tests.
dprophet Apr 3, 2023
22d8c8f
Fix TestSnowflakeConnectorTest.
yuuteng Apr 5, 2023
4a5f3c2
Fix type mapper
yuuteng Apr 6, 2023
6c95ead
Fix testconnector
yuuteng Apr 6, 2023
d5ac0dc
Update version to 413-SNAPSHOT
yuuteng Apr 6, 2023
4618db6
Merge branch 'snowflake' into bloomberg-snowflake
yuuteng Apr 6, 2023
8f79937
Merge branch 'bloomberg-snowflake' into feature/2551-add-snowflake-co…
yuuteng Apr 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .mvn/jvm.config
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED
--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
6 changes: 6 additions & 0 deletions core/trino-server/src/main/provisio/trino.xml
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,12 @@
</artifact>
</artifactSet>

<artifactSet to="plugin/snowflake">
<artifact id="${project.groupId}:trino-snowflake:zip:${project.version}">
<unpack />
</artifact>
</artifactSet>

<artifactSet to="plugin/sqlserver">
<artifact id="${project.groupId}:trino-sqlserver:zip:${project.version}">
<unpack />
Expand Down
1 change: 1 addition & 0 deletions docs/src/main/sphinx/connector.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ from different data sources.
Redis <connector/redis>
Redshift <connector/redshift>
SingleStore <connector/memsql>
Snowflake <connector/snowflake>
SQL Server <connector/sqlserver>
System <connector/system>
Thrift <connector/thrift>
Expand Down
88 changes: 88 additions & 0 deletions docs/src/main/sphinx/connector/snowflake.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
===================
Snowflake connector
===================

The Snowflake connector allows querying and creating tables in an
external Snowflake account. This can be used to join data between
different systems like Snowflake and Hive, or between two different
Snowflake accounts.

Configuration
-------------

To configure the Snowflake connector, create a catalog properties file
in ``etc/catalog`` named, for example, ``snowflake.properties``, to
mount the Snowflake connector as the ``snowflake`` catalog.
Create the file with the following contents, replacing the
connection properties as appropriate for your setup:

.. code-block:: none

connector.name=snowflake
connection-url=jdbc:snowflake://<account>.snowflakecomputing.com
connection-user=root
connection-password=secret
snowflake.account=account
snowflake.database=database
snowflake.role=role
snowflake.warehouse=warehouse


Multiple Snowflake databases or accounts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Snowflake connector can only access a single database within
a Snowflake account. Thus, if you have multiple Snowflake databases,
or want to connect to multiple Snowflake accounts, you must configure
multiple instances of the Snowflake connector.

.. snowflake-type-mapping:

Type mapping
------------

Trino supports the following Snowflake data types:

================================== ===============================
Snowflake Type Trino Type
================================== ===============================
``boolean`` ``boolean``
``tinyint`` ``bigint``
``smallint`` ``bigint``
``byteint`` ``bigint``
``int`` ``bigint``
``integer`` ``bigint``
``bigint`` ``bigint``
``float`` ``real``
``real`` ``real``
``double`` ``double``
``decimal`` ``decimal(P,S)``
``varchar(n)`` ``varchar(n)``
``char(n)`` ``varchar(n)``
``binary(n)`` ``varbinary``
``varbinary`` ``varbinary``
``date`` ``date``
================================== ===============================

Complete list of `Snowflake data types
<https://docs.snowflake.com/en/sql-reference/intro-summary-data-types.html>`_.

.. _snowflake-sql-support:

SQL support
-----------

The connector provides read access and write access to data and metadata in
a Snowflake database. In addition to the :ref:`globally available
<sql-globally-available>` and :ref:`read operation <sql-read-operations>`
statements, the connector supports the following features:

* :doc:`/sql/insert`
* :doc:`/sql/delete`
* :doc:`/sql/truncate`
* :doc:`/sql/create-table`
* :doc:`/sql/create-table-as`
* :doc:`/sql/drop-table`
* :doc:`/sql/alter-table`
* :doc:`/sql/create-schema`
* :doc:`/sql/drop-schema`
180 changes: 180 additions & 0 deletions plugin/trino-snowflake/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>io.trino</groupId>
<artifactId>trino-root</artifactId>
<version>413-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

<artifactId>trino-snowflake</artifactId>
<description>Trino - Snowflake Connector</description>
<packaging>trino-plugin</packaging>

<properties>
<air.main.basedir>${project.parent.basedir}</air.main.basedir>
</properties>

<dependencies>
<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-base-jdbc</artifactId>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-plugin-toolkit</artifactId>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>configuration</artifactId>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>log</artifactId>
yuuteng marked this conversation as resolved.
Show resolved Hide resolved
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
</dependency>

<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
</dependency>

<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<version>3.13.29</version>
</dependency>

<!-- Trino SPI -->
<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-spi</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>slice</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.openjdk.jol</groupId>
<artifactId>jol-core</artifactId>
<scope>provided</scope>
</dependency>

<!-- for testing -->
<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-base-jdbc</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-main</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-main</artifactId>
<type>test-jar</type>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-testing</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-testing-services</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino</groupId>
<artifactId>trino-tpch</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.trino.tpch</groupId>
<artifactId>tpch</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.airlift</groupId>
<artifactId>testing</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>jdbc</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>--add-opens=java.base/java.nio=ALL-UNNAMED</argLine>
</configuration>
</plugin>
</plugins>
</build>
</project>
Loading