Description

ErykCh

opened on Feb 14, 2024

Issue body actions

Is your feature request related to a problem?

Using LLM models provided by different companies with OpenAI API.

Describe the solution you'd like

For example there are LLM aggregators such as OpenRouter.
You can access it by OpenAI API. Here is an example:

client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=getenv("OPENROUTER_API_KEY"),
)

completion = client.chat.completions.create(
model="openchat/openchat-7b:free",
messages=[
{
"role": "user",
"content": "Say this is a test",
},
],
)

You are allow to provide api_key by OPENAI_API_KEY and base_url by OPENAI_API_ENDPOINT.
But there should be also flexible way to list available models so that you can use them later in the settings.

Maybe even OpenAI API allows to retrieve list of models and this would be awsome to list them.

Additional context

No response

Reacted by Benjamin Franklin

Activity

1. ErykCh

   added

   enhancementNew feature or requestNew feature or request

   on Feb 14, 2024
2. tianzhou commented on Feb 14, 2024

   

   tianzhou

   on Feb 14, 2024

   Member

   More actions

   Contribution welcome
3. tianzhou

   added

   help wantedExtra attention is neededExtra attention is needed

   on Feb 14, 2024
4. willkrakow commented on May 18, 2024

   

   willkrakow

   on May 18, 2024

   More actions

   Created a PR for this #154
5. benjamincburns commented on Jul 5, 2024

   

   benjamincburns

   on Jul 5, 2024

   More actions

   @ErykCh as a workaround, check out LiteLLM Proxy - you can configure it to map any model key string to any upstream API endpoint & model config that it can connect to. I'm not affiliated w/ that project, I've just found it useful for things like faking out the Zed editor's AI tooling to talk to a local Ollama (zed also restricts the model IDs you can assign, or at least it did last time I tried it), or for taking tooling that was written against the OpenAI client lib and pointing it at models on Bedrock.
6. jmorille commented on Nov 28, 2024

   

   jmorille

   on Nov 28, 2024

   More actions

   I have a similary issue using https://console.groq.com/docs/openai

   For Groq
   This offer OpeanAI compatibility and this model should be download like openai with the API

   • https://api.groq.com/openai/v1/models

   A quick look of your code, the model list is in static file: https://github.com/sqlchat/sqlchat/blob/main/src/utils/model.ts
   i am not familar with ReactJs, but it should be possible to replace by a APi call ..
   may be solve the APi comatiblity

   For Ollama , it should work in the same ways..
   cf https://github.com/ollama/ollama/blob/main/docs/openai.md
   Thank the OpenApi compatibilty feature

   Dynamic load models by API, This feature should be very welcome ..

Issue actions