First try app packaging : librechat - issue with postgresql extention (pgvector)

[nebulon]

Last time we checked, both could not be loaded in parallel. I am not sure if this is still the case after a quick research. But pgvectorS is supposed to be compatible with the other at least. Do you know more about this maybe or can you try with a patch to replace pgvector with pgvectors? Maybe this only a few seds

[Valexico]

Hmm I tried a sed and could create a vectors extension instead of vector

But then it cascaded to more errors. And with the different abstraction layers (langchain, sqlalchemy...) I couln't solve the issue.

I think current implementation tries to ceate a table with a VECTOR type anv vecto.rs expects VECTOR(dimension) (or is maybe not loaded properly)

The above exception was the direct cause of the following exception:
Nov 27 16:17:11 2024-11-27T15:17:11Z
Nov 27 16:17:11 Traceback (most recent call last):
Nov 27 16:17:11 File "/app/code/rag_api/main.py", line 46, in <module>
Nov 27 16:17:11 from psql import PSQLDatabase, ensure_custom_id_index_on_embedding, pg_health_check
Nov 27 16:17:11 File "/app/code/rag_api/psql.py", line 3, in <module>
Nov 27 16:17:11 from config import DSN, logger
Nov 27 16:17:11 File "/app/code/rag_api/config.py", line 232, in <module>
Nov 27 16:17:11 vector_store = get_vector_store(
Nov 27 16:17:11 File "/app/code/rag_api/store_factory.py", line 20, in get_vector_store
Nov 27 16:17:11 return AsyncPgVector(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/langchain_core/_api/deprecation.py", line 183, in warn_if_direct_instance
Nov 27 16:17:11 return wrapped(self, *args, **kwargs)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/langchain_community/vectorstores/pgvector.py", line 341, in __init__
Nov 27 16:17:11 self.__post_init__()
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/langchain_community/vectorstores/pgvector.py", line 355, in __post_init__
Nov 27 16:17:11 self.create_tables_if_not_exists()
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/langchain_community/vectorstores/pgvector.py", line 390, in create_tables_if_not_exists
Nov 27 16:17:11 Base.metadata.create_all(session.get_bind())
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/schema.py", line 5825, in create_all
Nov 27 16:17:11 bind._run_ddl_visitor(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 3254, in _run_ddl_visitor
Nov 27 16:17:11 conn._run_ddl_visitor(visitorcallable, element, **kwargs)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2460, in _run_ddl_visitor
Nov 27 16:17:11 visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/visitors.py", line 664, in traverse_single
Nov 27 16:17:11 return meth(obj, **kw)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/ddl.py", line 918, in visit_metadata
Nov 27 16:17:11 self.traverse_single(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/visitors.py", line 664, in traverse_single
Nov 27 16:17:11 return meth(obj, **kw)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/ddl.py", line 956, in visit_table
Nov 27 16:17:11 )._invoke_with(self.connection)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/ddl.py", line 314, in _invoke_with
Nov 27 16:17:11 return bind.execute(self)
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1421, in execute
Nov 27 16:17:11 return meth(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/sql/ddl.py", line 180, in _execute_on_connection
Nov 27 16:17:11 return connection._execute_ddl(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1532, in _execute_ddl
Nov 27 16:17:11 ret = self._execute_context(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1849, in _execute_context
Nov 27 16:17:11 return self._exec_single_context(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1989, in _exec_single_context
Nov 27 16:17:11 self._handle_dbapi_exception(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 2356, in _handle_dbapi_exception
Nov 27 16:17:11 raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/base.py", line 1970, in _exec_single_context
Nov 27 16:17:11 self.dialect.do_execute(
Nov 27 16:17:11 File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/engine/default.py", line 924, in do_execute
Nov 27 16:17:11 cursor.execute(statement, parameters)
Nov 27 16:17:11 sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject) type "vector" does not exist
Nov 27 16:17:11 LINE 4: embedding VECTOR,
Nov 27 16:17:11 ^
Nov 27 16:17:11 2024-11-27T15:17:11Z
Nov 27 16:17:11 [SQL:
Nov 27 16:17:11 CREATE TABLE langchain_pg_embedding (
Nov 27 16:17:11 collection_id UUID,
Nov 27 16:17:11 embedding VECTOR,
Nov 27 16:17:11 document VARCHAR,
Nov 27 16:17:11 cmetadata JSON,
Nov 27 16:17:11 custom_id VARCHAR,
Nov 27 16:17:11 uuid UUID NOT NULL,
Nov 27 16:17:11 PRIMARY KEY (uuid),
Nov 27 16:17:11 FOREIGN KEY(collection_id) REFERENCES langchain_pg_collection (uuid) ON DELETE CASCADE
Nov 27 16:17:11 )

[nebulon]

So looks like the application code needs to be adjusted further. I guess we have to see if both extensions can be made to work in parallel. So lets keep this as a feature request.

[nebulon]

We have managed to add the pgvector extension next to pgvectors. So far things are looking good, but we have to do more testing so we don't break other apps using postgres at the moment. But I am hopeful we can get this done with the next Cloudron release.

[Valexico]

Wow thanks @nebulon . I think it would have been necessary one day or another with the current AI hype.

By the way I'd be glad to share my "cloudron packaging prompt engineering" techniques some when. I'll make a second try with another app, I am thinking appsmith

[canadaduane]

This is awesome progress @Valexico! Did pgvector make it in to the base Cloudron release, and were you able to then connect LibreChat's DB up?

BTW I'm very interested in this right now because Open WebUI (the only chat frontend currently supported by Cloudrain AFAIU) recently changed their license to something that is no longer open source (by OSI definition).

[Valexico]

Hi @canadaduane
I think I did have a working app at the end but not 100% features. And then I didn't had time (and interest) to finish the setup
If somebody is interested I can share my past work but I guess litellm have change quite a bit since

[canadaduane]

I have a working setup, based on @Valexico 's work here:

github.com/canadaduane/librechat-cloudron
https://github.com/halecraft/librechat-cloudron

The package has been updated to the latest version of LibreChat 0.7.8, and I fixed a few minor wrinkles such as OIDC login, write access to the public dir, and postgres support. I haven't yet tested it with all of the providers (this requires access keys in the .env file).

[marcusquinn]

@canadaduane Amazing work, well done, and thanks @Valexico, too!

[girish]

@canadaduane great stuff! Is there any packaging issues to be aware of? Can you also attach some LICENSE to your work and we can then look into publishing it in the App Store.

[canadaduane]

Thanks @girish! I've added an MIT license. Also, I tested with a basic OPENROUTER_KEY and it seems to work well. I'm not aware of any issues at this time.

[canadaduane]

@Valexico is the MIT license ok with you also?

[Valexico]

@canadaduane
Yes no problem

[girish]

Thanks, cc @vladimir.d . We will take a look .

[firmansi]

It seems the package is not shipped with RAG API ?

[nebulon]

The package does have pgvector extension, that should work, however I am not sure I understand how the app needs to be configured for RAG API and the local vectordb. Does anyone have upstream docs for that? I can only find https://www.librechat.ai/docs/configuration/rag_api

[luckow]

more or less the same question here -> https://forum.cloudron.io/topic/14037/libcrechat-postgresql-pgvector

[firmansi]

@nebulon the documentation for RAG API can be found in https://github.com/danny-avila/rag_api

As seen that the pgvector can't be used if the RAG API not installed and running. This RAG API is somehow critical part, if it's not running then the librechat can't have capability that openwebui has for files upload, processing files to embedding model and so on

[nebulon]

I see, so this is actually another project which needs to be bundled together with librechat or as a separate app.

This one is using pgvector, not librechat itself!

(will also move the thread here to the librechat category)

[LoudLemur]

@Valexico hey there! Trying this is awesome and thank you so much for putting your ai skills towards packaging.

I came here to offer congratulations on having LibreChat packaged, but to hear you have stated an ai cloudron packaging initiative is just fantastic. I am very glad you are here and thanks for your efforts.