Advertisement

How to immediately cancel an Asyncio Task that uses the Ollama Python library to generate an answer?

阅读量:

题意: 如何立即取消使用Ollama Python库生成答案的Asyncio任务

问题背景:

Using Ollama's Python API, I can generate responses from large language models. My intention is to halt response generation by pressing a stop button. However, this method only works if a task has already initiated printing. If a task remains pending before printing begins, cancelling will have no effect—resulting in an unsolicited printed output. To clarify further, invoking prompt_mistral("Testing") continues execution despite my attempt to cancel.

我正通过Ollama的Python接口与大型语言模型(LLMs)互动以获取回答,并希望可以通过按下停止按钮来暂停当前运行中的请求处理过程。然而,在请求输出开始打印之前执行任务取消操作是有效的;但如果请求仍在执行中准备输出结果,则无法实现此功能——即使按下停止按钮,“prompt_mistral("Testing")” 该函数依然会被触发并输出结果。

My code: 我的代码

复制代码
 import ollama

    
 import asyncio
    
 import threading
    
 from typing import Optional
    
 import tkinter as tk
    
  
    
 # Create the main window
    
 root = tk.Tk()
    
 root.title("Tkinter Button Example")
    
  
    
 worker_loop: Optional[asyncio.AbstractEventLoop] = None
    
 task_future: Optional[asyncio.Future] = None
    
  
    
 async def get_answer_from_phi3():
    
  
    
     print("Trying")
    
  
    
     messages = [
    
         {"role": "system", "content": "Hello"}
    
     ]
    
  
    
     client = ollama.AsyncClient()
    
     
    
     stream = await client.chat(
    
     model='phi3',
    
     messages=messages,
    
     stream=True,
    
     options= {
    
     "top_k": 1})
    
  
    
     try:
    
     async for chunk in stream:
    
         # Store generated answer
    
        print(chunk['message']['content'], end='', flush=True)
    
         
    
  
    
     except asyncio.exceptions.CancelledError as e:
    
     print("Cancelled")
    
     pass
    
  
    
     except Exception as e:
    
     print(e)
    
     return "Sorry,vv an error occurred while processing your request."
    
  
    
  
    
 async def prompt_mistral(query):
    
     messages = []
    
     messages.append({"role": "assistant", "content": "Write a song that celebrates the beauty, diversity, and importance of our planet, Earth. The song should evoke vivid imagery of the natural world, from lush forests and majestic mountains to serene oceans and vast deserts. It should capture the essence of Earth as a living, breathing entity that sustains all forms of life. Incorporate themes of harmony, unity, and interconnectedness, emphasizing how all elements of nature are intertwined and how humanity is an integral part of this complex web. The lyrics should reflect a sense of wonder and appreciation for the planet's resources and ecosystems, highlighting the delicate balance that sustains life. Include references to various landscapes, climates, and wildlife, painting a picture of Earth's diverse environments. The song should also touch on the responsibility we have to protect and preserve the planet for future generations, addressing issues like climate change, deforestation, pollution, and conservation efforts. Use poetic language and metaphors to convey the grandeur and fragility of Earth, and infuse the song with a hopeful and inspiring tone that encourages listeners to take action in safeguarding our shared home. The melody should be uplifting and emotionally resonant, complementing the powerful message of the lyrics"})
    
     generated_answer = ''
    
     try:
    
     client = ollama.AsyncClient()
    
     stream = await client.chat(
    
         model='mistral',
    
         messages=messages,
    
         stream=True,
    
         options= {
    
             "top_k": 1}
    
     )
    
  
    
     async for chunk in stream:
    
         # Store generated answer
    
         generated_answer += chunk['message']['content']
    
         print(chunk['message']['content'])
    
         
    
     
    
     except asyncio.exceptions.CancelledError as e:
    
     print("Cancelled reponse")
    
     return
    
  
    
     except Exception as e:
    
     print(e)
    
  
    
     return "Sorry,vv an error occurred while processing your request."
    
  
    
 def prompt_llama(message):
    
  
    
     async def prompt():
    
  
    
     messages = []
    
     messages.append({"role": "assistant", "content": message})
    
     try:
    
         client = ollama.AsyncClient()
    
         stream = await client.chat(
    
             model='llama2',
    
             messages=messages,
    
             stream=True,
    
             options= {
    
                 "top_k": 1}
    
         )
    
  
    
         generated_answer = ''
    
         
    
         async for chunk in stream:
    
             # Store generated answer
    
             generated_answer += chunk['message']['content']
    
             print(chunk['message']['content'])    
    
     
    
         if "help" in generated_answer:
    
             await prompt_mistral("Testing")
    
         else:
    
             print(generated_answer)
    
  
    
     except asyncio.exceptions.CancelledError as e:
    
         print("Cancelled")
    
         return
    
  
    
     except Exception as e:
    
         print(e)
    
  
    
         return "Sorry,vv an error occurred while processing your request."
    
  
    
     def mistral_worker_function():
    
     global worker_loop, task_future
    
     worker_loop = asyncio.new_event_loop()
    
     task_future = worker_loop.create_task(prompt())
    
     worker_loop.run_until_complete(task_future) 
    
  
    
     print("Starting thread")
    
     thread = threading.Thread(target=mistral_worker_function)
    
     thread.start()
    
     client = ollama.AsyncClient()
    
  
    
 # Define the function to be called when the button is pressed
    
 def on_button_click():
    
     global worker_loop, task_future
    
     # the loop and the future are not threadsafe
    
     worker_loop.call_soon_threadsafe(
    
     lambda: task_future.cancel()
    
     )
    
     
    
  
    
     def phi3_worker_function():
    
     global worker_loop, task_future
    
     worker_loop = asyncio.new_event_loop()
    
     task_future = worker_loop.create_task(get_answer_from_phi3())
    
     worker_loop.run_until_complete(task_future)
    
  
    
     print("Starting thread")
    
     thread = threading.Thread(target=phi3_worker_function())
    
     thread.start()
    
  
    
 # Create the button
    
 button = tk.Button(root, text="Stop", command=on_button_click)
    
  
    
 # Place the button on the window
    
 button.pack(pady=20)
    
  
    
 prompt_llama("Hi")
    
  
    
 # Start the Tkinter event loop
    
 root.mainloop()

问题解决:

Upgrade the Ollama client to the latest version using the command $curl https://ollama.ai/install.sh | sh$ on Linux systems.

在Linux操作系统中,通过curl工具运行curl https://ollama.ai/install.sh | sh命令可以升级至最新版本的Ollama。

The system will automatically resolve this issue. The code operates precisely as intended. An update to Ollama is necessary.

它将自动修复这个问题。代码本身没有问题,只需要更新Ollama即可。

全部评论 (0)

还没有任何评论哟~