Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

running a model other than CoPilot crashes #5

Open
deven367 opened this issue Oct 2, 2023 · 6 comments
Open

running a model other than CoPilot crashes #5

deven367 opened this issue Oct 2, 2023 · 6 comments

Comments

@deven367
Copy link

deven367 commented Oct 2, 2023 

(fun) ➜  localpilot git:(main) ✗ python app.py
Running server...
DEBUG:asyncio:Using selector: KqueueSelector
INFO:     Started server process [8436]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:root:Running: python3 -m llama_cpp.server --model /Users/deven367/models/codellama-7b.Q5_K_S.gguf --n_gpu_layers 1 --n_ctx 4096
INFO:     127.0.0.1:49968 - "POST /set_target HTTP/1.1" 200 OK
Successfully sent selection: CodeLlama-7b.
DEBUG:root:Current state: {'url': 'https://huggingface.co/TheBloke/CodeLlama-7B-GGUF/resolve/main/codellama-7b.Q5_K_S.gguf', 'type': 'local', 'filename': 'codellama-7b.Q5_K_S.gguf'}
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/certifi/cacert.pem'
DEBUG:httpcore.connection:connect_tcp.started host='localhost' port=8000 local_address=None timeout=30 socket_options=None
DEBUG:httpcore.connection:connect_tcp.failed exception=ConnectError(OSError('All connection attempts failed'))
INFO:     127.0.0.1:49970 - "POST /v1/engines/copilot-codex/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
  | ExceptionGroup: multiple connection attempts failed (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('::1', 8000, 0, 0)
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 168, in try_connect
    |     stream = await asynclib.connect_tcp(remote_host, remote_port, local_address)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2202, in connect_tcp
    |     await get_running_loop().create_connection(
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
    |     raise exceptions[0]
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
    |     sock = await self._connect_sock(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
    |     await self.sock_connect(sock, address)
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
    |     return await fut
    |            ^^^^^^^^^
    |   File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
    |     raise OSError(err, f'Connect call failed {address}')
    | ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 8000)
    +------------------------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 114, in connect_tcp
    stream: anyio.abc.ByteStream = await anyio.connect_tcp(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/anyio/_core/_sockets.py", line 230, in connect_tcp
    raise OSError("All connection attempts failed") from cause
OSError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 366, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request
    response = await connection.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 99, in handle_async_request
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 76, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_backends/anyio.py", line 112, in connect_tcp
    with map_exceptions(exc_map):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/applications.py", line 116, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 746, in __call__
    await route.handle(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 55, in wrapped_app
    raise exc
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/_exception_handler.py", line 44, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/starlette/routing.py", line 70, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/projects/public/localpilot/proxy.py", line 65, in proxy
    r = await client.post(url, data=data, headers=headers, timeout=30)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1848, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 365, in handle_async_request
    with map_httpcore_exceptions():
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/deven367/mambaforge/envs/fun/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
@FGarridoV
Copy link

Same issue here.

@surak
Copy link

surak commented Oct 7, 2023

Seems like there should be another server running at port 8000, and it isn't.

@surak
Copy link

surak commented Oct 7, 2023

Okay, I tried running the model call which should open a 8000:

python3 -m llama_cpp.server --model '/Users/me/models/codellama-7b.Q5_K_S.gguf' --n_gpu_layers 1 --n_ctx 4096


llm_load_tensors: ggml ctx size =    0.09 MB
llm_load_tensors: mem required  = 4435.68 MB
..................................................................................................
llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: kv self size  = 2048.00 MB
llama_new_context_with_model: ggml_metal_init() failed
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/server/__main__.py", line 96, in <module>
    app = create_app(settings=settings)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/server/app.py", line 343, in create_app
    llama = llama_cpp.Llama(
            ^^^^^^^^^^^^^^^^
  File "/Users/surak/Devel/localpilot/venv/lib/python3.11/site-packages/llama_cpp/llama.py", line 377, in __init__
    assert self.ctx is not None
           ^^^^^^^^^^^^^^^^^^^^
AssertionError

So the error in my case is on llama.cpp on my machine.

@surak
Copy link

surak commented Oct 7, 2023

The way to see it is not hide the output of the process call for llama.cpp. Just edit the file proxy.py and change this:

diff --git a/proxy.py b/proxy.py
index 09d488c..177989b 100644
--- a/proxy.py
+++ b/proxy.py
@@ -22,7 +22,7 @@ def start_local_server(model_filename):
            "--n_gpu_layers", "1", "--n_ctx", "4096"]  # TODO: set this more correctly
     logging.debug('Running: %s' % ' '.join(cmd))
     local_server_process = subprocess.Popen(
-        cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+        cmd)#, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
 
 
 @app.route('/set_target', methods=['POST'])

@surak
Copy link

surak commented Oct 7, 2023

In my case (intel macbook pro), the error is that there's no "gpu" for it to use, so removing the n_gpu_layers from the llama.cpp call seems to work:

diff --git a/proxy.py b/proxy.py
index 09d488c..1e39b98 100644
--- a/proxy.py
+++ b/proxy.py
@@ -19,7 +19,7 @@ def start_local_server(model_filename):
         local_server_process.terminate()
         local_server_process.wait()
     cmd = ["python3", "-m", "llama_cpp.server", "--model", model_filename,
-           "--n_gpu_layers", "1", "--n_ctx", "4096"]  # TODO: set this more correctly
+        "--n_ctx", "4096"]  # TODO: set this more correctly
     logging.debug('Running: %s' % ' '.join(cmd))
     local_server_process = subprocess.Popen(
         cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

@surak
Copy link

surak commented Oct 7, 2023

Another way of testing your local llama.cpp is just opening http://localhost:8000/docs

@limdingwen limdingwen mentioned this issue Jan 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@surak @FGarridoV @deven367