Python module for on-the-fly Statistics

Overview

PyPi module https://pypi.org/project/on-the-fly-stats.html
git repository https://bitbucket.org/arrizza-public/on-the-fly-stats
git command git clone git@bitbucket.org:arrizza-public/on-the-fly-stats.git
Verification Report https://arrizza.com/web-ver/python-on-the-fly-stats-report.html
Version Info
  • macOS 14.2.1, Python 3.10
  • Ubuntu 20.04 focal, Python 3.10
  • Ubuntu 22.04 jammy, Python 3.10
  • win32 Windows 10, Python 3.10

Summary

This Python module contains various statistic functions (e.g. standard deviation) that can be updated on the fly i.e. the entire dataset is not needed up front.

Sample code

see sample.py for a full example

from on_the_fly_stats import OTFStats

stats = OTFStats()

# create an instance of the std deviation
stats.create_stddev('some_tag')
# add some values to it
stats.update_stddev('some_tag', 1)
stats.update_stddev('some_tag', 2)
stats.update_stddev('some_tag', 3)

# have other instances of it
stats.update_stddev('other_tag', 0.1)
stats.update_stddev('other_tag', 0.2)
stats.update_stddev('other_tag', 0.3)

# show the current data for a specific instance
print(stats.stddev['some_tag'].stddev)
print(stats.stddev['some_tag'].variance)
print(stats.stddev['some_tag'].mean)
print(stats.stddev['some_tag'].num_elements)

# show the current data for a specific instance
print(stats.stddev['other_tag'].stddev)

# some a summary report
stats.set_report_writer(print)
stats.report_minimal()
# skip the headers for even less data
stats.report_minimal(headers=False)

Min/Max

Holds the minimum and maximum of the values provided so far

import random

from on_the_fly_stats import OTFStats

stats = OTFStats()

stats.create_min_max('minmax1')
for _ in range(10):
    stats.update_min_max('minmax1', random.randint(0, 10))

print(stats.min_max['minmax1'].minimum)
print(stats.min_max['minmax1'].maximum)
print(stats.min_max['minmax1'].num_elements)

Average

Holds the average of the values provided so far

import random

from on_the_fly_stats import OTFStats

stats = OTFStats()
stats.create_average('avg1')
stats.update_average('avg1', random.uniform(0, 10))
stats.update_average('avg1', random.uniform(0, 10))

print(stats.average['avg1'].average)
print(stats.average['avg1'].anum_elementss)

Standard Deviation

Holds the standard deviation, mean and variance of the values provided so far

import random

from on_the_fly_stats import OTFStats

stats = OTFStats()
stats.create_stddev('sd1')
stats.update_stddev('sd1', random.random())
stats.update_stddev('sd1', random.random())
stats.update_stddev('sd1', random.random())

print(stats.stddev['sd1'].stddev)
print(stats.stddev['sd1'].mean)
print(stats.stddev['sd1'].variance)
print(stats.stddev['sd1'].num_elementss)

Counters

Holds counters of the values provided so far

from on_the_fly_stats import OTFStats

stats = OTFStats()

# automatically create a counter
stats.inc_counter('counter1')
stats.inc_counter('counter1')  # should be 2
stats.dec_counter('counter2')  # should be -1
stats.inc_counter('counter3')
stats.dec_counter('counter3')  # should be 0

print(stats.counters['counter1'].count)
print(stats.counters['counter2'].count)
print(stats.counters['counter3'].count)

Reports

There are two builtin reports:

  • a minimal report
  • a full report

To use the builtin reports a function that writes a single line must be provided. The simplest is to use print().

stats.set_report_writer(print)
stats.report_minimal()
stats.report()

Minimal report output

The minimal report shows each stat kept and minimal data with minimal formatting

Min/Max:
              0               7 minmax1
       0.019647        0.825751 minmax2
Average:
            4.495221 avg1
Stddev
     111.109474 stddev1
       0.300395 stddev2
Counters:
              2 counter1
             -1 counter2
              0 counter3

Full report output

The full report has more formatting and layout

---- Stats:

                 Min             Max statistic
     --------------- --------------- ------------------------------------------------------------------
                   0               7 minmax1
            0.019647        0.825751 minmax2
     >>> end of min/max

             Average statistic
     --------------- ------------------------------------------------------------------
            4.495221 avg1
     >>> end of Averages

              StdDev statistic
     --------------- ------------------------------------------------------------------
          111.109474 stddev1
            0.300395 stddev2
     >>> end of StdDev

               Total statistic
     --------------- ------------------------------------------------------------------
                   2 counter1
                  -1 counter2
                   0 counter3
     >>> end of counters

- John Arrizza