Skip to content

Commit

Permalink
updated documentation on chat endpoints and QvikChat guide
Browse files Browse the repository at this point in the history
  • Loading branch information
pranav-kural committed Aug 5, 2024
1 parent 1723dc3 commit ab07a38
Show file tree
Hide file tree
Showing 9 changed files with 118 additions and 102 deletions.
2 changes: 1 addition & 1 deletion pages/_meta.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"index": "Introduction",
"getting-started": "Getting Started",
"core": "Core",
"core": "QvikChat Guide",
"chat-endpoints": "Chat Endpoints",
"chat-agent": {
"display": "hidden"
Expand Down
46 changes: 24 additions & 22 deletions pages/chat-endpoints.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,45 +2,47 @@ import Img from "../components/img";

# Chat Endpoints

You can define various chat endpoints based on your requirements.
Each chat service is defined by defining a **chat endpoint**. The chat endpoint that you define, is the code that gets called when a new request is received at that endpoint. You can define multiple chat services, each with its own endpoint. Each chat endpoint can have various attributes like input data schema, LLM model to be used for processing queries, authentication policy, etc. You defines these attributes when creating the configurations for the chat endpoint.

For each chat endpoint you define, you can easily toggle features on and off. For example, you can enable chat history for any endpoint by simply setting `enableChatHistory` to `true`.
This is the real superpower of QvikChat. You can define a chat endpoint with support for chat history, authentication, response caching, RAG, and more, in just a few lines of code. You can even store these configurations remotely and load them dynamically at runtime.

It is this flexibility that QvikChat provides that enables you to quickly define custom chat endpoints with only a few lines of code.
There are two ways this can be done:

All chat endpoints can be defined using a single method: `defineChatEndpoint`. This method takes a configuration object as an argument, which allows you to define the chat endpoint based on your requirements.
1. **Using the `configureAndRunServer` method:** This is the easiest way to get started with QvikChat. You can provide a list of endpoint configurations to the `configureAndRunServer` method, and it will start the server with all the defined chat endpoints. To learn more about the `configureAndRunServer` method and the different configurations that you can provide, check the [Configure and Run Server](/core#configure-and-run-server) section.
2. **Using the `defineChatEndpoint` method:** You can directly call the `defineChatEndpoint` method with the configurations for the chat endpoint, anywhere in your codebase. This is the method that gets called under the hood when you provide the list of endpoint configurations to the `configureAndRunServer` method. When using the `defineChatEndpoint` method, you will have to manually ensure that you setup Firebase Genkit, and that all endpoints are defined before you start the server. To learn more about `defineChatEndpoint` method check the [Define Chat Endpoint](/core#define-chat-endpoint) section.

You can import the `defineChatEndpoint` method as shown below:
## Chat Endpoint Configurations

```typescript copy
import { defineChatEndpoint } from "@oconva/qvikchat/endpoints";
```
One of the most powerful features of the QvikChat framework is the flexibility and efficiency it provides in configuring chat endpoints. From chat history, response caching, and RAG to authentication, you can configure a chat endpoint with various features by simply specifying the configurations for the endpoint.

Below are the three types of chat agents you can define:
To learn more check [Chat Endpoint Configurations](/chat-endpoints/chat-endpoint-configurations).

## Open-Ended Chat
## Examples

You can checkout some examples of how to configure endpoints with various features like chat history, topic restrictions, context-awareness, and more.

To make it easier, these examples have been broadly categorized into three types:

### Open-Ended Chat

Unrestricted chat with no restrictions on what topic the user queries can be related to, quite similar to OpenAI's ChatGPT or Google's Gemini front-ends.

[Read more](chat-endpoints/open-ended-chat)
Check examples of [Open-ended chat](chat-endpoints/examples/open-ended-chat).

## Close-Ended Chat
### Close-Ended Chat

Chat endpoint where user queries are restricted to a specific topic. Prevents misuse of the chat service by restricting the usage, for example, a chat service meant to answer queries related to a specific topic like Firebase won't answer user queries related to other topics like solving a calculus assignment question.

[Read more](chat-endpoints/close-ended-chat)
Check examples of [Close-ended chat](chat-endpoints/examples/close-ended-chat)

## RAG Chat (Retrieval Augmented Generation)
### RAG Chat (Retrieval Augmented Generation)

Context-aware and topic-specific chat with ability to answer user queries by retrieving additional context information from a knowledge base (e.g. from a JSON file or PDF).

[Read more](chat-endpoints/rag-chat)
Check examples of [RAG chat](chat-endpoints/examples/rag-chat)

## Testing Endpoints

Below images gives a high-level overview of how a user query is handled in a RAG chat endpoint.
Once you have defined your chat endpoints, you can send queries to them through the Genkit Developer UI or by running your own server.

<Img
src="/images/rag-history-endpoint-query.png"
showSubtitle="true"
subtitle="High-level overview of how chat endpoints handle user queries"
priority={false}
/>
Learn more about [Testing Chat Endpoints](chat-endpoints/testing-endpoints).
4 changes: 1 addition & 3 deletions pages/chat-endpoints/_meta.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
{
"open-ended-chat": "Open-ended Chat",
"close-ended-chat": "Close-ended Chat",
"rag-chat": "RAG Chat",
"chat-endpoint-configurations": "Chat Endpoint Configurations",
"examples": "Examples",
"testing-endpoints": "Testing Endpoints"
}
99 changes: 55 additions & 44 deletions pages/chat-endpoints/chat-endpoint-configurations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Below are some of the chat endpoint configurations you can define.
<ModelConfigTable />
</details>

Please ensure that you have configured the project to use the model, if you specify any specific model name. For using models through the Gemini API or OpenAI API, ensure that you've setup the [correct environment variables](/getting-started#environment-variables). For any other models, please ensure you've added the Genkit plugin correctly. For more information on setting up Genkit plugins, check [Genkit integration](/integrations/genkit).

### Prompts

System prompt is used to configure the behavior, tone, and various other characteristics of the Large Language Model (LLM) model, before response generation. A well-structured system prompt designed with security and safety in mind can not only help generate high quality responses, it will also enable mitigation of LLM hallucinations and deterrence of malicious usage attempts (e.g., prompt injection attacks or LLM jailbreak attacks).
Expand Down Expand Up @@ -74,13 +76,8 @@ You can set the `verbose` property to `true` to get additional information in th

- `verbose`: If set to `true`, returns additional information in the response. May include usage information (like the number of input and output tokens used, input and output characters, etc.), tools calls information, and request details. By default, it is set to `false`. [Read more](#verbose-mode).

```typescript copy
defineChatEndpoint({
endpoint: "chat",
verbose: true,
});
```

<details>
<summary>More details about verbose output</summary>
The output produced by a chat endpoint where verbose is enabled, will contain an additional `details` object. This object may contain the following properties:

```typescript
Expand Down Expand Up @@ -119,45 +116,61 @@ type details = {
The details included in the `details` object come directly through [Firebase Genkit](https://github.com/firebase/genkit). The `usage` object contains information about the number of input and output tokens, characters, images, videos, audio files, and any custom data used in the response. The `tool_requests` object contains information about the tools called during the response generation. The `request` object contains the request details, including messages, data, etc.
## Response Type
</details>
You can set the `responseType` property to specify the type of response that the endpoint should return. The response type can be `text`, `json`, or `media`. By default, it is set to `text`.
## Multimodal I/O
```typescript copy
defineChatEndpoint({
endpoint: "chat",
responseType: "json",
});
```
- `outputSchema`: You can use this property to specify the expected data type of the response that the endpoint should return. This property expects an object with the following properties:
- `format`: The format of the response. This can be one of: `text`, `json`, or `media`. By default, it is set to `text`.
- `contentType`: The content type of the response. This is required if the `format` is set to `media`.
- `schema`: When the `format` is set to `json`, you can optionally provide the data schema that the response should adhere to. This schema can be specified using the [zod](https://www.npmjs.com/package/zod) library. Output from the LLM model will be validated against this schema before being returned.
The below code shows an example of how you can define the configurations for an image generation endpoint (using DALL·E 3 model) and an endpoint that returns JSON output.
<Notice>
Please note that, if you are using custom prompts with the endpoint, the
output schema of these prompts must match the response type that you configure
the endpoint with. Currently, `responseType` is available only in alpha
version and is still under testing. Presently, responses are returned only as
strings when using the default system and chat prompts. You can still get
responses back as media or JSON, but you will need to manually parse the
response.
output schema of these prompts must match the output schema that you configure
the endpoint with.
</Notice>
## Chat Agent Config

Under the hood, each chat endpoint uses a `ChatAgent` to process the query and generate responses. This chat agent has a LLM model specified for response generation, a default system prompt based on agent type, chat prompts, and optionally, any configured tools.

You can use the `chatAgentConfig` property to override the default configurations for the chat agent. Below are the properties you can set in the `chatAgentConfig` object:
```typescript
import { ChatEndpointConfig } from "@oconva/qvikchat";

// Configurations for the image generation endpoint (using DALL·E 3 model)
const imgUrlOutputEndpoint: ChatEndpointConfig = {
endpoint: "img",
modelConfig: {
name: "dallE3",
response_format: "url",
},
outputSchema: {
format: "media",
contentType: "image/png",
},
};

Please ensure that you have configured the project to use the model, if you specify any specific model name. For using models through the Gemini API or OpenAI API, ensure that you've setup the [correct environment variables](/getting-started#environment-variables). For any other models, please ensure you've added the Genkit plugin correctly. For more information on setting up Genkit plugins, check [Genkit integration](/integrations/genkit).
// Configurations for endpoint that returns JSON output
const jsonOutputEndpoint: ChatEndpointConfig = {
endpoint: "chat",
outputSchema: {
format: "json",
schema: z.object({
message: z.string(),
}),
},
};
```

## Example

Below is an example of a comprehensively configured chat endpoint. You don't need to provide all the configurations, only the ones you need. Below example is only for demonstration purposes.

```typescript copy
import { ChatEndpointConfig } from "@oconva/qvikchat";
import { FirestoreAPIKeyStore } from "@oconva/qvikchat/auth";
import { FirestoreCacheStore } from "@oconva/qvikchat/cache";
import { getEmbeddingModel } from "@oconva/qvikchat/data-embeddings";
import { TaskType } from "@oconva/qvikchat/data-retrievers";
import { defineChatEndpoint } from "@oconva/qvikchat/endpoints";
import { FirestoreChatHistoryStore } from "@oconva/qvikchat/history";
import { credential, getFirebaseApp } from "@oconva/qvikchat/firebase";

Expand All @@ -168,26 +181,24 @@ const firebaseApp = getFirebaseApp({
),
});

// Define chat endpoint with RAG, chat history, cache and auth
// Configurations for chat endpoint with RAG, chat history, cache and auth
// uses Firestore API key store, Firestore chat history store, Firestore cache store
// uses Gemini 15 Pro model for chat and embedding-001 embedding model
// uses custom retrieval strategy for RAG
defineChatEndpoint({
export const endpointConfig: ChatEndpointConfig = {
endpoint: "chat",
topic: "inventory data",
chatAgentConfig: {
model: "gemini15Pro",
modelConfig: {
version: "latest",
temperature: 0.5,
maxOutputTokens: 2048,
safetySettings: [
{
category: "HARM_CATEGORY_DANGEROUS_CONTENT",
threshold: "BLOCK_LOW_AND_ABOVE",
},
],
},
modelConfig: {
name: "gemini15Pro",
version: "latest",
temperature: 0.5,
maxOutputTokens: 2048,
safetySettings: [
{
category: "HARM_CATEGORY_DANGEROUS_CONTENT",
threshold: "BLOCK_LOW_AND_ABOVE",
},
],
},
enableAuth: true,
apiKeyStore: new FirestoreAPIKeyStore({
Expand Down Expand Up @@ -221,5 +232,5 @@ defineChatEndpoint({
taskType: TaskType.RETRIEVAL_DOCUMENT,
}),
},
});
};
```
5 changes: 5 additions & 0 deletions pages/chat-endpoints/examples/_meta.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"open-ended-chat": "Open-ended Chat",
"close-ended-chat": "Close-ended Chat",
"rag-chat": "RAG Chat"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

Chat endpoint where user queries are restricted to a specific topic. Prevents misuse of the chat service by restricting the usage, for example, a chat service meant to answer queries related to a specific topic like Firebase won't answer user queries related to other topics like solving a calculus assignment question.

You can define an close-ended chat endpoint with various features such as chat history, response caching, and API key authentication, by simply modifying the configurations provided to the `defineChatEndpoint` function. To use this, first import the method as following:
In QvikChat, you essentially write the configurations for a chat endpoint to specify the features you want to enable. To make it easier to write these configurations, we provide a TypeScript interface `ChatEndpointConfig` that you can use to define the chat endpoint configurations.

```typescript copy
import { defineChatEndpoint } from "@oconva/qvikchat/endpoints";
```typescript
import { ChatEndpointConfig } from "@oconva/qvikchat";
```

Below are some of the close-ended chat endpoints you can define.
Below are some examples of the close-ended chat endpoints you can define.

## Close Chat

Expand All @@ -21,11 +21,11 @@ Below is an example of how you can define a close-ended chat endpoint. You can u
```typescript copy
// Close-ended chat endpoint
// will only answer queries related to specified topic, in this case, 'Firebase'
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
endpoint: "chat-close",
agentType: "close-ended",
topic: "Firebase",
});
};
```

### Input Schema
Expand Down Expand Up @@ -94,13 +94,13 @@ Below is an example of how you can define a close-ended chat endpoint that suppo

```typescript copy copy {8}
// Close-ended chat endpoint with support for chat history
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
endpoint: "chat-close-history",
agentType: "close-ended",
topic: "Firebase",
enableChatHistory: true,
chatHistoryStore: new InMemoryChatHistoryStore(),
});
};
```

### Input Schema
Expand Down Expand Up @@ -183,7 +183,7 @@ testAPIKeyStore.addKey(key, {
});

// Close-ended chat endpoint with support for chat history, authentication, and caching
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
agentType: "close-ended",
topic: "Firebase",
endpoint: "chat-close-history-auth-cached",
Expand All @@ -193,7 +193,7 @@ defineChatEndpoint({
apiKeyStore: testAPIKeyStore,
enableCache: true,
cacheStore: new InMemoryCacheStore(),
});
};
```

### Input Schema
Expand Down Expand Up @@ -264,4 +264,4 @@ Endpoint to test: `chat-close-history-auth-cached`

## Configurations

As you can see from above, its quite easy to turn features on and off. For all chat endpoint configurations, please check [Chat Endpoint Configurations](/chat-endpoints/chat-flow-configurations).
As you can see from above, its quite easy to turn features on and off. For all chat endpoint configurations, please check [Chat Endpoint Configurations](/chat-endpoints/chat-endpoint-configurations).
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@

Unrestricted chat with no restrictions on what topic the user queries can be related to, quite similar to OpenAI's ChatGPT or Google's Gemini front-ends.

You can define an open-ended chat endpoint with various features such as chat history, response caching, and API key authentication, by simply modifying the configurations provided to the `defineChatEndpoint` function. To use this, first import the method as following:
In QvikChat, you essentially write the configurations for a chat endpoint to specify the features you want to enable. To make it easier to write these configurations, we provide a TypeScript interface `ChatEndpointConfig` that you can use to define the chat endpoint configurations.

```typescript
import { defineChatEndpoint } from "@oconva/qvikchat/endpoints";
import { ChatEndpointConfig } from "@oconva/qvikchat";
```

Below are some of the open-ended chat endpoints you can define.
Below are some examples of the open-ended chat endpoints you can define.

## Open Chat

Expand All @@ -22,9 +22,9 @@ Simply, pass the `endpoint` to the `defineChatEndpoint` function to define an op

```typescript copy
// Open-ended chat endpoint
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
endpoint: "chat-open",
});
};
```

### Input Schema
Expand Down Expand Up @@ -99,11 +99,11 @@ Below is an example of how you can define an open-ended chat endpoint that suppo
import { InMemoryChatHistoryStore } from "@oconva/qvikchat/history";

// Open-ended chat endpoint with support for chat history
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
endpoint: "chat-open-history",
enableChatHistory: true,
chatHistoryStore: new InMemoryChatHistoryStore(),
});
};
```

### Input Schema
Expand Down Expand Up @@ -190,15 +190,15 @@ testAPIKeyStore.addKey(key, {
});

// Open-ended chat endpoint with support for chat history, authentication, and caching
defineChatEndpoint({
const endpointConfig: ChatEndpointConfig = {
endpoint: "chat-open-history-auth-cached",
enableChatHistory: true,
chatHistoryStore: new InMemoryChatHistoryStore(),
enableAuth: true,
apiKeyStore: testAPIKeyStore,
enableCache: true,
cacheStore: new InMemoryCacheStore(),
});
};
```

### Input Schema
Expand Down Expand Up @@ -271,4 +271,4 @@ Endpoint to test: `chat-open-history-auth-cached`

## Configurations

As you can see from above, its quite easy to turn features on and off. For all chat endpoint configurations, please check [Chat Endpoint Configurations](/chat-endpoints/chat-flow-configurations).
As you can see from above, its quite easy to turn features on and off. For all chat endpoint configurations, please check [Chat Endpoint Configurations](/chat-endpoints/chat-endpoint-configurations).
Loading

0 comments on commit ab07a38

Please sign in to comment.