Adapt documentation, examples and demo scripts to Quarkus runtime

.gitignore

@@ -20,6 +20,7 @@
 regtests/derby.log
 regtests/metastore_db
 regtests/output/
+regtests/docker-compose.override.yml
 
 # Python stuff
 /poetry.lock

README.md

@@ -39,15 +39,36 @@ for contribution guidelines.
 ## Building and Running 
 
 Apache Polaris is organized into the following modules:
+
 - `polaris-core` - The main Polaris entity definitions and core business logic
-- `polaris-server` - The Polaris REST API server
-- `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface
+- API modules (implementing the Iceberg REST API and Polaris management API):
+  - `polaris-api-management-model` - The Polaris management model
+  - `polaris-api-management-service` - The Polaris management service
+  - `polaris-api-iceberg-service` - The Iceberg REST service
+- Service modules:
+  - `polaris-service-common` - The main components of the Polaris server
+- Quarkus runtime modules:
+  - `polaris-quarkus-service` - The Quarkus-specific components of the Polaris server
+  - `polaris-quarkus-server` - The Polaris server runtime
+  - `polaris-quarkus-admin-tool` - The Polaris admin & maintenance tool
+- Persistence modules
+  - `polaris-jpa-model` - The JPA entity definitions
+  - `polaris-eclipselink` - The Eclipselink implementation of the MetaStoreManager interface
 
 Apache Polaris is built using Gradle with Java 21+ and Docker 27+.
+
 - `./gradlew build` - To build and run tests. Make sure Docker is running, as the integration tests depend on it.
 - `./gradlew assemble` - To skip tests.
 - `./gradlew test` - To run unit tests and integration tests.
-- `./gradlew runApp` - To run the Polaris server locally on localhost:8181. 
+
+For local development, you can run the following commands:
+
+- `./gradlew --console=plain :polaris-quarkus-service:quarkusRun` - To run the Polaris server
+  locally, with profile `prod`; the server is reachable at localhost:8181.
+- `./gradlew --console=plain :polaris-quarkus-service:quarkusDev` - To run the Polaris server
+  locally, in [Quarkus Dev mode](https://quarkus.io/guides/dev-mode-differences). In dev mode, 
+  Polaris uses the `test` Authenticator and `test` TokenBroker; this configuration is suitable for 
+  running regressions tests, or for connecting with Spark.
 - `./regtests/run_spark_sql.sh` - To connect from Spark SQL. Here are some example commands to run in the Spark SQL shell:
 ```sql
 create database db1;
@@ -57,36 +78,76 @@ insert into db1.table1 values (1, 'a');
 select * from db1.table1;
 ```
 
-Apache Polaris supports the following optional build options:
-- `-PeclipseLink=true` – Enables the EclipseLink extension.
-- `-PeclipseLinkDeps=[groupId]:[artifactId]:[version],...` – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.
-
 ### More build and run options
-Running in Docker
-- `docker build -t localhost:5001/polaris:latest .` - To build the image.
-  - Optional build options:
-    - `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true .` - Enables the EclipseLink extension.
-    - `docker build -t localhost:5001/polaris:latest --build-arg ECLIPSELINK=true --build-arg ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],... .` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas.
-- `docker run -p 8181:8181 localhost:5001/polaris:latest` - To run the image in standalone mode.
-
-Running in Kubernetes
-- `./run.sh` - To run Polaris as a mini-deployment locally. This will create one pod that bind itself to ports `8181` and `8182`.
-  - Optional run options:
-    - `./run.sh -b "ECLIPSELINK=true"` - Enables the EclipseLink extension.
-    - `./run.sh -b "ECLIPSELINK=true;ECLIPSELINK_DEPS=[groupId]:[artifactId]:[version],..."` – Enables the EclipseLink extension with one or more additional dependencies for EclipseLink (e.g. JDBC drivers) separated by commas.
-- `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure connections between a local machine and a pod within the cluster for both service and metrics endpoints.
-  - Currently supported metrics endpoints:
-    - localhost:8182/metrics
-    - localhost:8182/healthcheck
+
+#### Running in Docker
+
+Please note: there are no official Docker images for Apache Polaris yet. For now, you can build the
+Docker images locally.
+
+To build the Polaris server Docker image locally:
+
+```shell
+./gradlew clean :polaris-quarkus-server:assemble -Dquarkus.container-image.build=true
+```
+
+To run the Polaris server Docker image:
+
+```shell
+docker run -p 8181:8181 -p 8182:8182 apache/polaris:latest
+```
+
+#### Running in Kubernetes
+
+- `./run.sh` - To run Polaris as a mini-deployment locally. This will create a Kind cluster, 
+  then deploy one pod and one service. The service is available on ports `8181` and `8182`.
+- `kubectl port-forward svc/polaris-service -n polaris 8181:8181 8182:8182` - To create secure 
+  connections between a local machine and a pod within the cluster for both service and metrics 
+  endpoints.
+  - Currently supported metrics and health endpoints:
+    - http://localhost:8182/q/metrics
+    - http://localhost:8182/q/health
 - `kubectl get pods -n polaris` - To check the status of the pods.
 - `kubectl get deployment -n polaris` - To check the status of the deployment.
 - `kubectl describe deployment polaris-deployment -n polaris` - To troubleshoot if things aren't working as expected.
 
-Running regression tests
-- `./regtests/run.sh` - To run regression tests in another terminal.
-- `docker compose up --build --exit-code-from regtest` - To run regression tests in a Docker environment.
+#### Running regression tests
+
+Regression tests can be run in a local environment or in a Docker environment.
+
+To run regression tests locally, you need to have a Polaris server running locally, with the 
+`test` Authenticator enabled. You can do this by running Polaris in Quarkus Dev mode, as explained
+above:
+
+```shell
+./gradlew --console=plain :polaris-quarkus-service:quarkusDev
+```
+
+Then, you can run the regression tests using the following command:
+
+```shell
+./regtests/run.sh
+```
+
+To run regression tests in a Docker environment, you can use the following command:
+
+```shell
+docker compose -f regtests/docker-compose.yml up --build --exit-code-from regtest
+```
+
+The above command will by default run Polaris with the Docker image `apache/polaris:latest`; if you
+want to use a different image, you can modify the `docker-compose.yaml` file prior to running it;
+alternatively, you can use the following commands to override the image:
+
+```shell
+cat <<EOF > regtests/docker-compose.override.yml
+services: { polaris: { image: localhost:5001/apache/polaris:latest } }
+EOF
+docker compose -f regtests/docker-compose.yml up --build --exit-code-from regtest
+```
+
+#### Building docs
 
-Building docs
 - Docs are generated using [Hugo](https://gohugo.io/) using the [Docsy](https://www.docsy.dev/docs/) theme.
 - To view the site locally, run
   ```bash

k8/deployment.yaml

@@ -39,7 +39,7 @@ spec:
     spec:
       containers:
       - name: polaris
-        image: localhost:5001/polaris:latest
+        image: localhost:5001/apache/polaris:latest
         ports:
         - containerPort: 8181
         - containerPort: 8182

quarkus/admin/src/main/resources/application.properties

@@ -24,4 +24,4 @@ quarkus.container-image.push=false
 quarkus.container-image.registry=docker.io
 quarkus.container-image.group=apache
 quarkus.container-image.name=polaris-admin-tool
-quarkus.container-image.additional-tags=latest
+quarkus.container-image.additional-tags=latest

quarkus/service/src/main/resources/application.properties

@@ -77,7 +77,7 @@ polaris.context.realm-context-resolver.header-name=Polaris-Realm
 polaris.context.realm-context-resolver.type=default
 
 polaris.features.defaults."ENFORCE_PRINCIPAL_CREDENTIAL_ROTATION_REQUIRED_CHECKING"=false
-polaris.features.defaults."SUPPORTED_CATALOG_STORAGE_TYPES"=["S3","GCS","AZURE","FILE"]
+polaris.features.defaults."SUPPORTED_CATALOG_STORAGE_TYPES"=["S3","GCS","AZURE"]
 # realm overrides
 # polaris.features.realm-overrides."my-realm"."INITIALIZE_DEFAULT_CATALOG_FILEIO_FOR_TEST"=true
 # polaris.features.realm-overrides."my-realm"."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION"=true
@@ -148,3 +148,17 @@ polaris.authentication.token-broker.max-token-generation=PT1H
 %test.polaris.storage.aws.secret-key=secretKey
 %test.polaris.storage.gcp.token=token
 %test.polaris.storage.gcp.lifespan=PT1H
+
+%dev.quarkus.log.file.enable=false
+%dev.quarkus.log.category."org.apache.polaris".level=DEBUG
+%dev.quarkus.log.category."org.apache.iceberg.rest".level=DEBUG
+%dev.quarkus.otel.sdk.disabled=true
+%dev.polaris.authentication.authenticator.type=test
+%dev.polaris.authentication.token-service.type=test
+%dev.polaris.features.defaults."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE","S3","GCS","AZURE"]
+%dev.polaris.context.realm-context-resolver.default-realm=POLARIS
+%dev.polaris.persistence.type=in-memory
+%dev.polaris.storage.aws.access-key=accessKey
+%dev.polaris.storage.aws.secret-key=secretKey
+%dev.polaris.storage.gcp.token=token
+%dev.polaris.storage.gcp.lifespan=PT1H

regtests/docker-compose.yml

@@ -35,6 +35,7 @@ services:
       polaris.persistence.type: in-memory
       polaris.authentication.authenticator.type: test
       polaris.authentication.token-service.type: test
+      polaris.features.defaults."SUPPORTED_CATALOG_STORAGE_TYPES": '["FILE","S3","GCS","AZURE"]'
       quarkus.otel.sdk.disabled: "true"
     volumes:
       - ./credentials:/tmp/credentials/

run.sh

@@ -21,26 +21,16 @@
 
 # Runs Polaris as a mini-deployment locally. Creates two pods that bind themselves to port 8181.
 
-# Initialize variables
-BUILD_ARGS=""  # Initialize an empty string to store Docker build arguments
-
 # Function to display usage information
 usage() {
-  echo "Usage: $0 [-b build-arg1=value1;build-arg2=value2;...] [-h]"
-  echo "  -b    Pass a set of arbitrary build arguments to docker build, separated by semicolons"
+  echo "Usage: $0 [-h]"
   echo "  -h    Display this help message"
   exit 1
 }
 
 # Parse command-line arguments
 while getopts "b:h" opt; do
   case ${opt} in
-    b)
-      IFS=';' read -ra ARGS <<< "${OPTARG}"  # Split the semicolon-separated list into an array
-      for arg in "${ARGS[@]}"; do
-        BUILD_ARGS+=" --build-arg ${arg}"  # Append each build argument to the list
-      done
-      ;;
     h)
       usage
       ;;
@@ -57,20 +47,17 @@ shift $((OPTIND-1))
 echo "Building Kind Registry..."
 sh ./kind-registry.sh
 
-# Check if BUILD_ARGS is not empty and print the build arguments
-if [[ -n "$BUILD_ARGS" ]]; then
-  echo "Building polaris image with build arguments:$BUILD_ARGS"
-else
-  echo "Building polaris image without any additional build arguments."
-fi
-
 # Build and deploy the server image
 echo "Building polaris image..."
-docker build -t localhost:5001/polaris $BUILD_ARGS -f Dockerfile .
+./gradlew :polaris-quarkus-server:build \
+  -Dquarkus.container-image.build=true \
+  -Dquarkus.container-image.registry=localhost:5001
+
 echo "Pushing polaris image..."
-docker push localhost:5001/polaris
+docker push localhost:5001/apache/polaris
+
 echo "Loading polaris image to kind..."
-kind load docker-image localhost:5001/polaris:latest
+kind load docker-image localhost:5001/apache/polaris:latest
 
 echo "Applying kubernetes manifests..."
 kubectl delete -f k8/deployment.yaml --ignore-not-found

site/content/in-dev/unreleased/admin-tool.md

@@ -0,0 +1,97 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+linkTitle: Polaris Admin Tool
+title: Apache Polaris (Incubating) Admin Tool
+type: docs
+weight: 300
+---
+
+In order to help administrators manage their Polaris database, Polaris provides an administration
+tool.
+
+The tool is built using [Quarkus](https://quarkus.io/).
+
+## How to Download the Admin Tool
+
+As of January 2025, there is currently no binary release or official Docker image available for
+the tool. For now, you need to build the artifacts yourself, for example, by running the following
+command:
+
+```shell
+./gradlew :polaris-quarkus-admin:assemble -Dquarkus.container-image.build=true
+```
+
+The above command will generate:
+
+- One standalone JAR in `quarkus/admin/build/polaris-quarkus-admin-*-runner.jar`
+- One Docker image `apache/polaris-server-admin-tool:latest`
+- One Docker image `apache/polaris-server-admin-tool:<version>`
+
+## Usage
+
+To run the standalone JAR, use the following command:
+
+```shell
+java -jar quarkus/admin/build/polaris-quarkus-admin-*-runner.jar --help
+```
+
+To run the Docker image, use the following command:
+
+```shell
+docker run -p 8181:8181 apache/polaris-server-admin-tool:latest --help
+```
+`
+The basic usage of the Polaris Admin Tool is outlined below:
+
+```
+Usage: polaris-server-admin-tool.jar [-hV] [COMMAND]
+Polaris Admin Tool
+  -h, --help      Show this help message and exit.
+  -V, --version   Print version information and exit.
+Commands:
+  help       Display help information about the specified command.
+  bootstrap  Bootstraps principal credentials.
+  purge      Purge principal credentials.
+```
+
+## Configuration
+
+The Polaris Admin Tool uses the same configuration as the Polaris server. The configuration can be
+done via environment variables or system properties.
+
+At a minimum, it is necessary to configure the Polaris Admin Tool to connect to the same database
+used by the Polaris server. This can be done by setting the following system properties:
+
+```shell
+java -jar quarkus/admin/build/polaris-quarkus-admin-*-runner.jar \
+    -Dpolaris.persistence.eclipselink.configuration-file=/path/to/persistence.xml
+    -Dpolaris.persistence.eclipselink.persistence-unit=polaris
+```
+
+See the [metastore documentation]({{% ref "metastores" %}}) for more information on configuring the
+database connection.
+
+## Bootstrapping Principal Credentials
+
+TODO
+
+## Purging Principal Credentials
+
+TODO