Skip to content

Commit

Permalink
fix: Bq deployment, docs update with correct BQ variables for deploym…
Browse files Browse the repository at this point in the history 
…ent (#10)

* feat: Adding BQ deployment model and restructuring directory

* feat: polishing up documentation for bq deployment; adding cloud shell options for both deployment styles

* Removing dist folders

* Removing dist from git cache

* Adding tags for lookml metadata and adjusting prompt

* fix: updating docs with appropriate BQ variable names in examples; adding open in colab link

---------

Co-authored-by: Luka Fontanilla <maluka@google.com>
  • Loading branch information
@LukaFontanilla
LukaFontanilla and Luka Fontanilla authored Mar 25, 2024
1 parent 0208fa2 commit 7afd555
Show file tree
Hide file tree
Showing 6 changed files with 24 additions and 20 deletions.
5 changes: 3 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,9 @@ curl --location 'http://localhost:8000/' --header 'Content-Type: application/jso
}'
```

Before deploying, you will want to swap out the example LookML explore metadata text file and jsonl file for files that are customized for your Explore data. Please
see [this Colab Notebook](./explore-assistant-training/looker_explore_assistant_training.ipynb) to automate this.
Before deploying, you will want to swap out the example LookML explore metadata text file and jsonl file for files that are customized for your Explore data. Please see this notebook for more details on generating these: <a target="_blank" href="https://colab.research.google.com/github/LukaFontanilla/looker-explore-assistant/blob/main/explore-assistant-training/looker_explore_assistant_training.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

### 2. Deployment

Expand Down
28 changes: 15 additions & 13 deletions explore-assistant-extension/extension-bigquery-deployment/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,26 +48,29 @@ Terraform deployment coming soon!
gcloud projects add-iam-policy-binding 'PROJECT_NUMBER' --member='serviceAccount:MEMBER' --role='roles/aiplatform.user' --condition=None
```
6. From either the BigQuery console or your Looker Instance's SQL Runner Tab, execute the following commands:
Create a remote model connection to `gemini-pro`. Please replace the following variables with those for you environment (`PROJECT_ID`,`DATASET_ID`,`REGION`)
```sql
CREATE OR REPLACE MODEL
`PROJECT_ID.DATASET_ID.explore_assistant_llm`
REMOTE WITH CONNECTION `PROJECT_ID.REGION.explore_assistant`
OPTIONS (ENDPOINT = 'gemini-pro');
```
Create a dataset in your BigQuery instance called `explore_assistant`. Please replace the following variables with those for you environment (`PROJECT_ID`,`REGION`)
```sql
CREATE SCHEMA IF NOT EXISTS `PROJECT_ID.explore_assistant`
OPTIONS(location = 'REGION')
```
Create a remote model connection to `gemini-pro`. Please replace the following variables with those for you environment (`PROJECT_ID`,`REGION`). The remote model and connection will be named
the same (`explore_assistant_llm`) for simplicity.
```sql
CREATE OR REPLACE MODEL
`PROJECT_ID.explore_assistant.explore_assistant_llm`
REMOTE WITH CONNECTION `PROJECT_ID.REGION.explore_assistant_llm`
OPTIONS (ENDPOINT = 'gemini-pro');
```
Execute the following command to create a table in that dataset called `explore_assistant_examples` to hold the examples for each explore. This will serve as a lookup table.
```sql
CREATE OR REPLACE TABLE `explore_assistant.explore_assistant_examples` (
explore_id STRING OPTIONS (description = 'Explore id of the explore to pull examples for in a format of -> lookml_model:lookml_explore'),
examples STRING OPTIONS (description = 'Examples for Explore Assistant training. Multi line string of input: ,output: \n')
)
```
Copy Stringified Examples and set them as variable in BQ. (**NOTE:** Examples below are standard ecomm examples, you will need to copy in those relevant to your environment. Please see [this notebook for more details on generating these](../../explore-assistant-training/looker_explore_assistant_training.ipynb))
Copy Stringified Examples and set them as variable in BQ. (**NOTE:** Examples below are standard ecomm examples used to illustrate the structure, you will need to copy in those relevant to your environment. Please see this notebook for more details on generating these: <a target="_blank" href="https://colab.research.google.com/github/LukaFontanilla/looker-explore-assistant/blob/main/explore-assistant-training/looker_explore_assistant_training.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
```sql
DECLARE examples STRING;
SET examples = """
Expand All @@ -80,16 +83,15 @@ Terraform deployment coming soon!
input: Orders that are still in Processing after 3 days, filtered by Distribution Center
output:fields=order_items.created_date,order_items.order_id,products.item_name,order_items.status,users.email,order_items.average_days_to_process&f[distribution_centers.name]=Chicago IL&f[order_items.created_date]=before 3 days ago&f[order_items.status]=Processing&sorts=order_items.created_date desc&column_limit=50&vis={\"type\":\"looker_grid\"}"""
```
Insert these examples with the Looker explore id (ie. model:explore) into the `explore_assistant` table.
Insert these examples with the Looker explore id (ie. model:explore; replacing "model" with the lookml model and "explore" with the lookml explore) into the `explore_assistant` table.
```sql
INSERT INTO explore_assistant_demo_logs.explore_assistant_examples (explore_id,examples)
INSERT INTO explore_assistant.explore_assistant_examples (explore_id,examples)
VALUES ('model:explore',examples);
```
)
### 2. Looker Extension Framework Setup
**Important** If you are not familiar with the Looker Extension Framework, please review [this documentation](https://developers.looker.com/extensions/overview/) first before moving forward.
Expand Down Expand Up @@ -119,7 +121,7 @@ Terraform deployment coming soon!
```
1. Start the development server
**IMPORTANT** If you are running the extension from a VM or another remote machine, you will need to Port Forward to the machine where you are accessing the Looker Instance from. Here's a boilerplate example for port forwarding the remote port 8080 to the local port 8080:
**IMPORTANT** If you are running the extension from a VM or another remote machine, you will need to Port Forward to the machine where you are accessing the Looker Instance from (ie. if you are accessing Looker from your local machine, run the following command there). Here's a boilerplate example for port forwarding the remote port 8080 to the local port 8080:
`ssh username@host -L 8080:localhost:8080`.

```bash
Expand All @@ -142,7 +144,7 @@ Terraform deployment coming soon!
```lookml
application: explore-assistant {
label: "Explore Assistant"
# file: "explore-assistant.js"
# file: "bundle.js"
url: "https://localhost:8000/bundle.js"
entitlements: {
core_api_methods: ["lookml_model_explore","run_inline_query","create_sql_query","run_sql_query"]
Expand Down
4 changes: 2 additions & 2 deletions explore-assistant-extension/extension-bigquery-deployment/src/helpers.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,14 +68,14 @@ const generateText = (request: GenerateTextRequest) => {

const insertResponse = (request: GenerateTextFeedback) => {
return `
INSERT INTO explore_assistant_demo.explore_assistant_responses (explore_id,input,output,accurate,feedback)
INSERT INTO explore_assistant.explore_assistant_responses (explore_id,input,output,accurate,feedback)
VALUES ('${request.model}:${request.explore}','${request.input}','${request.response}',,)
`
}

const insertResponseFeedback = (request: GenerateTextFeedback) => {
return `
UPDATE explore_assistant_demo.explore_assistant_responses SET accurate = ${request.accurate} AND feedback = '${request.feedback}'
UPDATE explore_assistant.explore_assistant_responses SET accurate = ${request.accurate} AND feedback = '${request.feedback}'
WHERE explore_id = '${request.model}:${request.explore}'
`
}
Expand Down
2 changes: 1 addition & 1 deletion explore-assistant-extension/extension-cloud-function-deployment/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ The deployed Cloud Function will be public by default. This is because historica
```

1. Start the development server
**IMPORTANT** If you are running the extension from a VM or another remote machine, you will need to Port Forward to the machine where you are accessing the Looker Instance from. Here's a boilerplate example for port forwarding the remote port 8080 to the local port 8080:
**IMPORTANT** If you are running the extension from a VM or another remote machine, you will need to Port Forward to the machine where you are accessing the Looker Instance from (ie. If you are accessing Looker from your local machine, run the following command there.). Here's a boilerplate example for port forwarding the remote port 8080 to the local port 8080:
`ssh username@host -L 8080:localhost:8080`.

```bash
Expand Down
3 changes: 2 additions & 1 deletion ...assistant-extension/extension-cloud-function-deployment/cloud-function/src/examples.jsonl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@
{"input": "customers who likes columbia or levi's", "output": "fields=users.email,products.brand,order_items.total_sale_price&f[products.brand]=Columbia, Levi's&sorts=order_items.total_sale_price desc 0&limit=500"}
{"input": "Last week's revenue by category and department", "output": "fields=products.category,products.department,order_items.total_sale_price&pivots=products.department&order_items.created_year&f[order_items.created_date]=last week&sorts=order_items.total_sale_price desc 0&limit=500&column_limit=50"}
{"input": "Sales performance by state, on a map", "output": "fields=order_items.order_count,users.count,order_items.total_sale_price,order_items.average_spend_per_user,users.state&f[order_items.created_date]=90 days&sorts=order_items.total_sale_price desc&limit=500&column_limit=50&vis={\"type\" "}
{"input": "Who are the customer with highest revenue in New York?", "output": "fields=users.email,user_order_facts.lifetime_revenue&f[users.state]=New York&sorts=user_order_facts.lifetime_revenue desc 0&limit=500=vis_config={\"type\" "}
{"input": "Who are the customer with highest revenue in New York?", "output": "fields=users.email,user_order_facts.lifetime_revenue&f[users.state]=New York&sorts=user_order_facts.lifetime_revenue desc 0&limit=500=vis_config={\"type\" "}
{"input": "Show monthly profit for the last year. Pivot on product category.Include only jeans vs. accessories.Display as an area chart.","output": "fields=products.category,order_items.total_gross_margin,order_items.created_month_name&pivots=products.category&f[products.category]=Jeans, Accessories&f[order_items.created_date]=last year&sorts=order_items.created_month_name asc,order_items.total_gross_margin desc 0&limit=500&vis={\"type\":\"looker_area\"}"}
2 changes: 1 addition & 1 deletion explore-assistant-training/looker_explore_assistant_training.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
"source": [
"# Explore Assistant Examples Generation\n",
"\n",
"This notebook is a companion to the [Explore Assistant Looker + GenAI Solution](https://sites.google.com/corp/google.com/pblv2/looker-x-genai?authuser=0) and will take you through some example code for:\n",
"This notebook is a companion to the [Explore Assistant Looker + GenAI Solution](https://github.com/LukaFontanilla/looker-explore-assistant) and will take you through some example code for:\n",
"\n",
"\n",
"* Formatting Looker Explore Metadata for Prompt Stuffing\n",
Expand Down

0 comments on commit 7afd555

Please sign in to comment.