ISRI/ai-server
10
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

added a lock to avoid two inference at the same time and added consequent support for asynchronous generator based model

Browse source
This commit is contained in:
faraphel 2025-01-10 19:11:48 +01:00
parent c6d779f591
commit 775c78c6cb
4 changed files with 19 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

5
samples/models/dummy/model.py
View file

@ -1,4 +1,3 @@
import json
import typing
@ -8,5 +7,5 @@ def load(model) -> None:
def unload(model) -> None:
pass
def infer(model, file) -> typing.Iterator[bytes]:
yield json.dumps({"hello": "world!"}).encode("utf-8")
async def infer(model, file) -> typing.AsyncIterator[bytes]:
yield await file.read()