Overview
| module url | https://pypi.org/project/async-url-caller-base.html | ||||||||||
| git repository | https://bitbucket.org/arrizza-public/async-url-caller-base | ||||||||||
| git command | git clone git@bitbucket.org:arrizza-public/async-url-caller-base.git | ||||||||||
| verification report | https://arrizza.com/web-ver/async-url-caller-base-report.html | ||||||||||
| version info |
|
- copyright: Copyright and License
- repo status: Repo Information
- installation: Common Setup
Summary
A python base class for calling URLs on a web server in parallel.
I use this to double-check if my website's valid pages are accessible (e.g. check-valid-pages and check-sitemap).
I also use it to check for any incoming URLs are correctly handled by web-server's nginx. For example, in the last 15 days it had 33,000 calls, it tested all of those in 23 seconds.
- See
sample/app.pyfor some code that you can run usingzpm doit. Note this uses the unit test server inut/helper/ut_server.py. - See
doc/usage-scenarios.mdfor common scenarios - See
doc/api.mdfor a list of functions.
It can handle normal HTTP GET URLs and it can also handle, what I've called "raw" URLs e.g. those using "CONNECT", "PRI", "MKCOL", or those with other formats specific to your server.
Why a Base Class?
In my experience, this kind of code is needed in a variety of scenarios. So a base class gives substantial flexibility in extending the behavior available.
Also, the behavior tends to be a part of a larger utility. A base class allows it to be embedded much easier into that larger code base.
Common usage scenarios
Under Construction
GET/POST URLs only
"raw" URLs
Both "get" and "raw"
API documentation
Under Construction
TODO top level
AsyncUrlCallerBase
Common functions
run_all_tests(urls, cb_get=None, cb_raw=None, need_content=False, allow_redirects=True, is_https=True)- xxcb_get(url, info, response, content=None)- xxcb_raw(url, info, response, content=None)- xxversion- (property) current versionthread_multiplier- (property) sets the thread multiplier; default: 3
caller = AsyncUrlCallerBase()
caller.thread_multiplier = 2
# if the PC has 12 CPUs, then there will be 24 threads used
TODO
set_raw_parms(ip_address, port)- xxset_timeouts(max_loops=None, max_timeout=None, timeouts=None, delay=None)- xx
Callbacks - handle exceptions
set_excp_general_cb(cb_func)- xxset_excp_timeout_error_cb(cb_func)- xxset_excp_client_error_cb(cb_func)- xxset_excp_reset_error_cb(cb_func)- xxset_unknown_status_code_cb(cb_func)- xx
Overridable
is_excluded_get(url)- xxis_excluded_raw(url)- xx
fn_name- (property) desc
sample usage