Feat: BQ Deployment (#7)

* feat: Adding BQ deployment model and restructuring directory

* feat: polishing up documentation for bq deployment; adding cloud shell options for both deployment styles

* Removing dist folders

* Removing dist from git cache

* Adding tags for lookml metadata and adjusting prompt

---------

Co-authored-by: Luka Fontanilla <maluka@google.com>

.gitignore

@@ -3,12 +3,20 @@
 temp_db.json
 .env
 .eslintcache
-explore-assistant-extension/cloud-function/terraform/.terraform
-explore-assistant-extension/cloud-function/terraform/terraform.tfstate
-explore-assistant-extension/cloud-function/terraform/terraform.tfstate.backup
-explore-assistant-extension/cloud-function/terraform/.terraform.lock.hcl
-explore-assistant-extension/dist
-explore-assistant-extension/node_modules
+explore-assistant-extension/extension-cloud-function-deployment/cloud-function/terraform/.terraform
+explore-assistant-extension/extension-cloud-function-deployment/cloud-function/terraform/terraform.tfstate
+explore-assistant-extension/extension-cloud-function-deployment/cloud-function/terraform/terraform.tfstate.backup
+explore-assistant-extension/extension-cloud-function-deployment/cloud-function/terraform/.terraform.lock.hcl
+explore-assistant-extension/extension-cloud-function-deployment/node_modules
+explore-assistant-extension/extension-cloud-function-deployment/dist/
+explore-assistant-extension/extension-cloud-function-deployment/.env
+explore-assistant-extension/extension-bigquery-deployment/bigquery/terraform/.terraform
+explore-assistant-extension/extension-bigquery-deployment/bigquery/terraform/terraform.tfstate
+explore-assistant-extension/extension-bigquery-deployment/bigquery/terraform/terraform.tfstate.backup
+explore-assistant-extension/extension-bigquery-deployment/bigquery/terraform/.terraform.lock.hcl
+explore-assistant-extension/extension-bigquery-deployment/node_modules
+explore-assistant-extension/extension-bigquery-deployment/dist/
+explore-assistant-extension/extension-bigquery-deployment/.env
 explore-assistant-api/explore-assistant
 explore-assistant-api/explore-assistant/
 explore-assistant-api/terraform/.terraform

README.md

@@ -12,9 +12,10 @@ Additionally, the extension provides:
 
  - Question History (*this is stored in the browser with IndexDB*)
  - Categorized Prompts (*these can be customized by the use cases of your organization*)
- - **NEW** Cached Explore URL's when clicking from History
- - **NEW** Structured Logging with Input & Output Token Counts (*enables a workflow of log sink to BQ for cost estimation & tracking*)
- - **NEW** Gemini Pro Update in Cloud Function
+ - Cached Explore URL's when clicking from History
+ - Structured Logging with Input & Output Token Counts (*enables a workflow of log sink to BQ for cost estimation & tracking*)
+ - Gemini Pro Update in Cloud Function
+ - **NEW** BigQuery Deployment Workflow
 
 Upcoming capabilities on the roadmap:
 
@@ -41,175 +42,30 @@ Upcoming capabilities on the roadmap:
 - ---
 
 ## Setup Explore Assistant Extension
-### 1. Generative AI Endpoint
+There are two options for deployment this repository provides, please choose the deployment model that fits your use case:
+* [Cloud Function Deployment](./explore-assistant-extension/extension-cloud-function-deployment/README.md): This deployment model is great if you are an application developer and plan on either leveraging an LLM deployed on infrastructure outside of the Vertex AI Platform or using the Looker + LLM integration as a standalone API in your own application.
+* [BigQuery Deployment](./explore-assistant-extension//extension-bigquery-deployment/README.md): This deployment model is great if you want to manage training data and the integration itself through BigQuery and plan on using a model deployed on Vertex AI Platform infrastructure.
 
-This section describes how to set up the Gen AI endpoint for the Explore Assistant. TLDR; We use a 2nd Gen Cloud Function to call the foundational model and return the results to the frontend.
-
-#### Getting Started for Development
-
-![simple-architecture](./static/simple-architecture.png)
-
-1. Clone or download a copy of this repository to your development machine.
-
-   ```bash
-   # cd ~/ Optional. your user directory is usually a good place to git clone to.
-   git clone git@github.com:LukaFontanilla/looker-explore-assistant.git
-   ```
-
-2. Navigate (`cd`) to the template directory on your system
-
-   ```bash
-   cd looker-explore-assistant/explore-assistant-extension/cloud-function/terraform
-   ```
-
-3. Replace defaults in the `variables.tf` file for project and region.
-
-4. Deploy resources.
-
-   ```terraform
-   terraform init
-
-   terraform plan
-
-   terraform apply
-   ```
-
-5. Save Deployed Cloud Function URL Endpoints
-
-#### Optional: Deploy regional endpoints and load balance traffic from Looker
-
-![global-architecture](./static/global-architecture.png)
-
-Please see this resource for more information on how to deploy regional endpoints and load balance traffic from Looker: https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless
-
-#### Optional: Setup Log Sink to BQ for LLM Cost Estimation and Request Logging
-
-Please see [Google Cloud's docs](https://cloud.google.com/logging/docs/export/configure_export_v2#creating_sink) on setting up a log sink to BQ, using the below filter for Explore Assistant Logs:
-
-```
-(resource.type = "cloud_function"
-resource.labels.function_name = "Insert service name"
-resource.labels.region = "<Insert location>")
- OR 
-(resource.type = "cloud_run_revision"
-resource.labels.service_name = "<Insert service name>"
-resource.labels.location = "<Insert location>")
- severity>=DEFAULT
-jsonPayload.component="explore-assistant-metadata"
-```
-
-
-
-### 2. Looker Extension Framework Setup
-
-
-#### Getting Started for Development
-
-1. Navigate (`cd`) to the template directory on your system
-
-   ```bash
-   cd looker-explore-assistant/explore-assistant-extension
-   ```
-
-1. Install the dependencies with [NPM](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
-
-   ```bash
-   npm install
-   ```
-
-   > You may need to update your Node version or use a [Node version manager](https://github.com/nvm-sh/nvm) to change your Node version.
-
-1. Ensure all the appropriate environment variables are set.
-
-   ```
-   VERTEX_AI_ENDPOINT=
-   LOOKER_MODEL=
-   LOOKER_EXPLORE=
-   ```
-
-1. Start the development server
-
-   ```bash
-   npm start
-   ```
-
-   Great! Your extension is now running and serving the JavaScript at https://localhost:8080/bundle.js.
-
-1. Now log in to Looker and create a new project.
-
-   This is found under **Develop** => **Manage LookML Projects** => **New LookML Project**.
-
-   You'll want to select "Blank Project" as your "Starting Point". You'll now have a new project with no files.
-
-   1. In your copy of the extension project you have a `manifest.lkml` file.
-
-   You can either drag & upload this file into your Looker project, or create a `manifest.lkml` with the same content. Change the `id`, `label`, or `url` as needed.
-
-   ```lookml
-   application: explore_assistant {
-    label: "Explore Assistant"
-    url: "https://localhost:8080/bundle.js"
-    # file: "bundle.js"
-    entitlements: {
-      core_api_methods: ["lookml_model_explore"]
-      navigation: yes
-      use_embeds: yes
-      use_iframes: yes
-      new_window: yes
-      new_window_external_urls: ["https://developers.generativeai.google/*"]
-      local_storage: yes
-      external_api_urls: ["cloud function url"]
-    }
-   }
-   ```
-
-1. Create a `model` LookML file in your project. The name doesn't matter. The model and connection won't be used, and in the future this step may be eliminated.
-
-   - Add a connection in this model. It can be any connection, it doesn't matter which.
-   - [Configure the model you created](https://docs.looker.com/data-modeling/getting-started/create-projects#configuring_a_model) so that it has access to some connection.
-
-1. Connect your new project to Git. You can do this multiple ways:
-
-   - Create a new repository on GitHub or a similar service, and follow the instructions to [connect your project to Git](https://docs.looker.com/data-modeling/getting-started/setting-up-git-connection)
-   - A simpler but less powerful approach is to set up git with the "Bare" repository option which does not require connecting to an external Git Service.
-
-1. Commit your changes and deploy your them to production through the Project UI.
-
-1. Reload the page and click the `Browse` dropdown menu. You should see your extension in the list.
-   - The extension will load the JavaScript from the `url` provided in the `application` definition. By default, this is https://localhost:8080/bundle.js. If you change the port your server runs on in the package.json, you will need to also update it in the manifest.lkml.
-   - Refreshing the extension page will bring in any new code changes from the extension template, although some changes will hot reload.
-
-#### Deployment
-
-The process above requires your local development server to be running to load the extension code. To allow other people to use the extension, a production build of the extension needs to be run. As the kitchensink uses code splitting to reduce the size of the initially loaded bundle, multiple JavaScript files are generated.
-
-1. In your extension project directory on your development machine, build the extension by running the command `npm build`.
-2. Drag and drop ALL of the generated JavaScript files contained in the `dist` directory into the Looker project interface.
-3. Modify your `manifest.lkml` to use `file` instead of `url` and point it at the `bundle.js` file.
-
-Note that the additional JavaScript files generated during the production build process do not have to be mentioned in the manifest. These files will be loaded dynamically by the extension as and when they are needed. Note that to utilize code splitting, the Looker server must be at version 7.21 or above.
-
----
-
-# [Optional] Setup Looker Explore Assistant API
+## [Optional] Setup Looker Explore Assistant API
 ## Description
 The Explore Assistant API is an API only version of the Explore Assistant intended to be integrated with your Backend to surface visualizations from natural language in a custom application. Below the requirements, setup and an example curl request is detailed:
 
 ### 1. Setup Explore Assistant API and Run Locally
 
+From the root of the directory (ie. `looker-explore-assistant/`)
 ```bash
-cd looker-explore-assistant/explore-assistant-api &&
+cd explore-assistant-api &&
 export PROJECT=<your GCP Project> &&
 export REGION=<your GCP region>
 ```
 
 Ensure you have `venv` installed before running the following:
 ```bash
 # Create a new virtualenv named "explore-assistant"
-python3 -m venv explore-assistant
+python3 -m venv explore-assistant &&
 
 # Activate the virtualenv (OS X & Linux)
-source explore-assistant/bin/activate
+source explore-assistant/bin/activate &&
 
 # Activate the virtualenv (Windows)
 explore-assistant\Scripts\activate
@@ -230,21 +86,28 @@ see [this Colab Notebook](./explore-assistant-training/looker_explore_assistant_
 
 ### 2. Deployment
 
-1. Navigate (`cd`) to the template directory on your system
+1. From the Explore Assistant root directory (`cd`) to the Explore Assistant API folder.
 
    ```bash
-   cd looker-explore-assistant/explore-assistant-api/terraform
+   cd explore-assistant-api/terraform
    ```
 
 3. Replace defaults in the `variables.tf` file for project, region and endpoint name.
 
-4. Deploy resources.
+4. Ensure that [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) is installed on your machine. Then Deploy resources.
 
+   Initialize Terraform
    ```terraform
    terraform init
+   ```
 
+   Plan resource provisioning
+   ```
    terraform plan
+   ```
 
+   Provision Resources
+   ```
    terraform apply
    ```
 

explore-assistant-extension/extension-bigquery-deployment/.env_example

@@ -0,0 +1,3 @@
+LOOKER_EXPLORE=<This is your Looker Explore Name>
+LOOKER_MODEL=<This is your Looker Model Name>
+BQML_MODEL_ID=<This is your BQML model id in the format dataset.modelname>