Overview

module url https://pypi.org/project/async-url-caller-base.html
git repository https://bitbucket.org/arrizza-public/async-url-caller-base
git command git clone git@bitbucket.org:arrizza-public/async-url-caller-base.git
verification report https://arrizza.com/web-ver/async-url-caller-base-report.html
version info
OS Language #Runs Last Run Cov%
Ubuntu 26.04 resolute Python 3.12 930 2026-11-15 93%

Summary

A python base class for calling URLs on a web server in parallel.

I use this to double-check if my website's valid pages are accessible (e.g. check-valid-pages and check-sitemap).

I also use it to check for any incoming URLs are correctly handled by web-server's nginx. For example, in the last 15 days it had 33,000 calls, it tested all of those in 23 seconds.

  • See sample/app.py for some code that you can run using zpm doit. Note this uses the unit test server in ut/helper/ut_server.py.
  • See doc/usage-scenarios.md for common scenarios
  • See doc/api.md for a list of functions.

It can handle normal HTTP GET URLs and it can also handle, what I've called "raw" URLs e.g. those using "CONNECT", "PRI", "MKCOL", or those with other formats specific to your server.

Why a Base Class?

In my experience, this kind of code is needed in a variety of scenarios. So a base class gives substantial flexibility in extending the behavior available.

Also, the behavior tends to be a part of a larger utility. A base class allows it to be embedded much easier into that larger code base.


Common usage scenarios

Under Construction

GET/POST URLs only
"raw" URLs

Both "get" and "raw"


API documentation

Under Construction

TODO top level

AsyncUrlCallerBase

Common functions

  • run_all_tests(urls, cb_get=None, cb_raw=None, need_content=False, allow_redirects=True, is_https=True) - xx
  • cb_get(url, info, response, content=None) - xx
  • cb_raw(url, info, response, content=None) - xx
  • version - (property) current version
  • thread_multiplier - (property) sets the thread multiplier; default: 3
caller = AsyncUrlCallerBase()
caller.thread_multiplier = 2
# if the PC has 12 CPUs, then there will be 24 threads used

TODO

  • set_raw_parms(ip_address, port) - xx
  • set_timeouts(max_loops=None, max_timeout=None, timeouts=None, delay=None) - xx

Callbacks - handle exceptions

  • set_excp_general_cb(cb_func) - xx
  • set_excp_timeout_error_cb(cb_func) - xx
  • set_excp_client_error_cb(cb_func) - xx
  • set_excp_reset_error_cb(cb_func) - xx
  • set_unknown_status_code_cb(cb_func) - xx

Overridable

  • is_excluded_get(url) - xx
  • is_excluded_raw(url) - xx

  • fn_name - (property) desc
sample usage

- John Arrizza