Asked  7 Months ago    Answers:  5   Viewed   42 times

I'm getting the flow of using asyncio in Python 3.5 but I haven't seen a description of what things I should be awaiting and things I should not be or where it would be neglible. Do I just have to use my best judgement in terms of "this is an IO operation and thus should be awaited"?

 Answers

23

By default all your code is synchronous. You can make it asynchronous defining functions with async def and "calling" these functions with await. A More correct question would be "When should I write asynchronous code instead of synchronous?". Answer is "When you can benefit from it". In cases when you work with I/O operations as you noted you will usually benefit:

# Synchronous way:
download(url1)  # takes 5 sec.
download(url2)  # takes 5 sec.
# Total time: 10 sec.

# Asynchronous way:
await asyncio.gather(
    async_download(url1),  # takes 5 sec. 
    async_download(url2)   # takes 5 sec.
)
# Total time: only 5 sec. (+ little overhead for using asyncio)

Of course, if you created a function that uses asynchronous code, this function should be asynchronous too (should be defined as async def). But any asynchronous function can freely use synchronous code. It makes no sense to cast synchronous code to asynchronous without some reason:

# extract_links(url) should be async because it uses async func async_download() inside
async def extract_links(url):  

    # async_download() was created async to get benefit of I/O
    html = await async_download(url)  

    # parse() doesn't work with I/O, there's no sense to make it async
    links = parse(html)  

    return links

One very important thing is that any long synchronous operation (> 50 ms, for example, it's hard to say exactly) will freeze all your asynchronous operations for that time:

async def extract_links(url):
    data = await download(url)
    links = parse(data)
    # if search_in_very_big_file() takes much time to process,
    # all your running async funcs (somewhere else in code) will be frozen
    # you need to avoid this situation
    links_found = search_in_very_big_file(links)

You can avoid it calling long running synchronous functions in separate process (and awaiting for result):

executor = ProcessPoolExecutor(2)

async def extract_links(url):
    data = await download(url)
    links = parse(data)
    # Now your main process can handle another async functions while separate process running    
    links_found = await loop.run_in_executor(executor, search_in_very_big_file, links)

One more example: when you need to use requests in asyncio. requests.get is just synchronous long running function, which you shouldn't call inside async code (again, to avoid freezing). But it's running long because of I/O, not because of long calculations. In that case, you can use ThreadPoolExecutor instead of ProcessPoolExecutor to avoid some multiprocessing overhead:

executor = ThreadPoolExecutor(2)

async def download(url):
    response = await loop.run_in_executor(executor, requests.get, url)
    return response.text
Tuesday, June 1, 2021
 
LaKaede
answered 7 Months ago
44

You have already called loop.close() before you ran that sample piece of code, on the global event loop:

>>> import asyncio
>>> asyncio.get_event_loop().close()
>>> asyncio.get_event_loop().is_closed()
True
>>> asyncio.get_event_loop().run_until_complete(asyncio.sleep(1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete
    self._check_closed()
  File "/.../lib/python3.6/asyncio/base_events.py", line 357, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

You need to create a new loop:

loop = asyncio.new_event_loop()

You can set that as the new global loop with:

asyncio.set_event_loop(asyncio.new_event_loop())

and then just use asyncio.get_event_loop() again.

Alternatively, just restart your Python interpreter, the first time you try to get the global event loop you get a fresh new one, unclosed.

As of Python 3.7, the process of creating, managing, then closing the loop (as well as a few other resources) is handled for you when use asyncio.run(). It should be used instead of loop.run_until_complete(), and there is no need any more to first get or set the loop.

Thursday, July 15, 2021
 
koenHuybrechts
answered 5 Months ago
99

When should one implement a service?

When you have work -- delivering value to the user -- that:

  • Needs some time to complete, perhaps longer than you have time for in the component wishing the work to be done, or

  • Is delivering that value under user control (e.g., music player, controlled by play/pause buttons in a UI), or

  • In rare cases, needs to be running continuously, as it delivers value continuously

there are quite a lot of them running and I doubt it's just a poor application design

Some are likely to be poor implementations, either due to technical misunderstandings, or other concerns (e.g., making marketing happy) trumping making users happy.

It was a background sound recorder and I was using it as Foreground service with notification since I wanted buttons to be able to control it (like music players do for example)

That is a reasonable use for a service, IMHO.

Intent listeners are quite a useful feature and using them as you know is usually enough for many use-cases (for example showing notifications)

I assume that by "Intent listeners" you mean manifest-registered BroadcastReceivers. In that case, if the work to be done by the BroadcastReceiver will take more than a millisecond, that work should be delegated to an IntentService for completion. onReceive() is called on the main application thread, and it is not safe for a manifest-registered BroadcastReceiver to fork a bare thread, as the process could go away shortly after onReceive() returns. However, in these cases, the service is usually short-lived (e.g., do some network I/O and disk I/O, then go away).

In the development documentation services are suggested for network communication and background calculations but I don't get why you should not just use AsyncTasks for that

An AsyncTask is a fine solution for background work that is:

  • Requested by the UI (activity or fragment), and

  • Will take less than a second or so, and

  • Is non-critical

For example, if you are downloading avatars to show in a ListView, AsyncTask is probably a fine choice, whether you use them directly or use some image-fetching library that uses them internally.

Conversely, if the user buys an MP3 through your app, and you need to download that MP3 file, an AsyncTask is not a good solution. That could easily take over a second. While the download is going on, the user could switch away from the app (e.g., press HOME). At that point, your process is eligible to be terminated... perhaps before your download is complete. Using an IntentService to manage the download is a signal to the OS that you are really doing work here, adding value to the user, and so the process will be left alone for a little while.

Note that if the background work might take 15+ seconds, WakefulBroadcastReceiver or my WakefulIntentService is probably a good idea, so the device does not fall asleep while you are trying to wrap up this bit of work.

Saturday, September 4, 2021
 
d8aninja
answered 3 Months ago
94

First, you should consider using loop.run_in_executor with a ProcessPoolExecutor if you plan to run python subprocesses from within the loop. As for your problem, you can use the event loop policy functions to set a new loop:

import asyncio
from concurrent.futures import ProcessPoolExecutor

async def sub_main():
    print('Hello from subprocess')

def sub_loop():
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(sub_main())

async def start(executor):
    await asyncio.get_event_loop().run_in_executor(executor, sub_loop)

if __name__ == '__main__':
    executor = ProcessPoolExecutor()
    asyncio.get_event_loop().run_until_complete(start(executor))
Wednesday, September 29, 2021
 
Stubbi
answered 2 Months ago
38

If txt is already equal to a jquery object, there is no need to use $(txt) as it's just extra processing to return the same thing.

Saturday, November 13, 2021
 
RLH
answered 3 Weeks ago
RLH
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :  
Share