TorCtl.StatsSupport
index
/data/users/mikeperry/code/SOAT/torflow-trunk-clean/TorCtl/StatsSupport.py

Support classes for statisics gathering
 
The StatsSupport package contains several classes that extend
PathSupport to gather continuous statistics on the Tor network.
 
The main entrypoint is to extend or create an instance of the
StatsHandler class. The StatsHandler extends from
TorCtl.PathSupport.PathBuilder, which is itself a TorCtl.EventHandler.
The StatsHandler listens to CIRC and STREAM events and gathers all
manner of statics on their creation and failure before passing the
events back up to the PathBuilder code, which manages the actual
construction and the attachment of streams to circuits.
 
The package also contains a number of support classes that help gather
additional statistics on the reliability and performance of routers.
 
For the purpose of accounting failures, the code tracks two main classes
of failure: 'actual' failure and 'suspected' failure. The general rule
is that an actual failure is attributed to the node that directly
handled the circuit or stream. For streams, this is considered to be the
exit node. For circuits, it is both the extender and the extendee.
'Suspected' failures, on the other hand, are attributed to every member
of the circuit up until the extendee for circuits, and all hops for
streams.
 
For bandwidth accounting, the average stream bandwidth and the average
ratio of stream bandwidth to advertised bandwidth are tracked, and when
the statistics are written, a Z-test is performed to calculate the
probabilities of these values assuming a normal distribution. Note,
however, that it has not been verified that this distribution is
actually normal. It is likely to be something else (pareto, perhaps?).

 
Modules
       
TorCtl.PathSupport
TorCtl.TorCtl
TorCtl.TorUtil
copy
math
random
re
sys
time
traceback

 
Classes
       
TorCtl.PathSupport.PathBuilder(TorCtl.TorCtl.ConsensusTracker)
StatsHandler
BandwidthStats
ReasonRouterList
FailedRouterList
SuspectRouterList
TorCtl.TorCtl.Router
StatsRouter

 
class BandwidthStats
    Class that manages observed bandwidth through a Router
 
  Methods defined here:
__init__(self)
add_bw(self, bytes, duration)
Add an observed transfer of 'bytes' for 'duration' seconds

 
class FailedRouterList(ReasonRouterList)
    Helper class to track all routers that failed for a given
reason. The main difference between this and the normal
ReasonRouterList is the sort order and the verification.
 
  Methods defined here:
__init__(self, reason)
sort_list(self)

Methods inherited from ReasonRouterList:
add_r(self, r)
Add a router to the list for this reason
total_failed(self)
Get a list of total failures for this reason
total_suspected(self)
Get a list of total suspected failures for this reason
write_list(self, f)
Write the list of failure counts for this reason 'f'

 
class ReasonRouterList
    Helper class to track which Routers have failed for a given reason
 
  Methods defined here:
__init__(self, reason)
add_r(self, r)
Add a router to the list for this reason
sort_list(self)
total_failed(self)
Get a list of total failures for this reason
total_suspected(self)
Get a list of total suspected failures for this reason
write_list(self, f)
Write the list of failure counts for this reason 'f'

 
class StatsHandler(TorCtl.PathSupport.PathBuilder)
    An extension of PathSupport.PathBuilder that keeps track of 
router statistics for every circuit and stream
 
 
Method resolution order:
StatsHandler
TorCtl.PathSupport.PathBuilder
TorCtl.TorCtl.ConsensusTracker
TorCtl.TorCtl.EventHandler
TorCtl.TorCtl.EventSink

Methods defined here:
__init__(self, c, slmgr, RouterClass=<class TorCtl.StatsSupport.StatsRouter at 0xa7a4398c>, track_ranks=False)
avg_adv_bw(self)
avg_circ_failure(self)
avg_circ_suspects(self)
avg_stream_failure(self)
avg_stream_suspects(self)
circ_status_event(self, c)
close_circuit(self, id)
count_stream_reason_failed(self, s, reason)
Count the routers involved in a failure
count_stream_suspects(self, s, lreason, reason)
Count the routers 'suspected' of being involved in a failure
new_consensus_event(self, n)
new_desc_event(self, d)
reset(self)
reset_stats(self)
run_zbtest(self)
Run unweighted z-test to calculate the probabilities of a node
having a given stream bandwidth based on the Normal distribution
run_zrtest(self)
Run unweighted z-test to calculate the probabilities of a node
having a given ratio of stream bandwidth to advertised bandwidth
based on the Normal distribution
stream_status_event(self, s)
write_ratios(self, filename)
Write out bandwith ratio stats StatsHandler has gathered
write_reasons(self, f, reasons, name)
Write out all the failure reasons and statistics for all Routers
write_routers(self, f, rlist, name)
Write out all the usage statistics for all Routers
write_stats(self, filename)
Write out all the statistics the StatsHandler has gathered

Data and other attributes defined here:
ratio_key = 'Metatroller Ratio Statistics:\n SR=Stream avg ra...eam Count\n P=Percentile Rank U=Uptime (h)\n'

Methods inherited from TorCtl.PathSupport.PathBuilder:
attach_stream_any(self, stream, badcircs)
Attach a stream to a valid circuit, avoiding any in 'badcircs'
bandwidth_event(self, b)
build_path(self)
Get a path from the SelectionManager's PathSelector, can be used 
e.g. for generating paths without actually creating any circuits
circuit_list(self)
Return an iterator or a list of circuits prioritized for 
stream selection.
close_all_circuits(self)
Close all open circuits
close_all_streams(self, reason)
Close all open streams
heartbeat_event(self, event)
This function handles dispatching scheduled jobs. If you 
extend PathBuilder and want to implement this function for 
some reason, be sure to call the parent class
schedule_immediate(self, job)
Schedules an immediate job to be run before the next event is
processed.
schedule_low_prio(self, job)
Schedules a job to be run when a non-time critical event arrives.
schedule_selmgr(self, job)
Schedules an immediate job to be run before the next event is
processed. Also notifies the selection manager that it needs
to update itself.
stream_bw_event(self, s)

Data and other attributes inherited from TorCtl.PathSupport.PathBuilder:
is_urgent_event = <TorCtl.TorUtil.Callable instance at 0x82637ac>

Methods inherited from TorCtl.TorCtl.ConsensusTracker:
current_consensus(self)
update_consensus(self)

Methods inherited from TorCtl.TorCtl.EventHandler:
add_event_listener(self, evlistener)
address_mapped_event(self, event)
Called when Tor adds a mapping for an address if listening
to ADDRESSMAPPED events.
buildtimeout_set_event(self, event)
guard_event(self, event)
msg_event(self, event)
Called when a log message of a given severity arrives if listening
to INFO_MSG, NOTICE_MSG, WARN_MSG, or ERR_MSG events.
ns_event(self, event)
or_conn_status_event(self, event)
Called when an OR connection's status changes if listening to
ORCONNSTATUS events.
timer_event(self, event)
unknown_event(self, event)
Called when we get an event type we don't recognize.  This
is almost alwyas an error.

 
class StatsRouter(TorCtl.TorCtl.Router)
    Extended Router to handle statistics markup
 
  Methods defined here:
__init__(self, router)
'Promotion Constructor' that converts a Router directly into a 
StatsRouter without a copy.
__str__(self)
adv_ratio(self)
Return the ratio of the Router's advertised bandwidth to 
the overall average observed bandwith
avg_extend_time(self)
Return the average amount of time it took for this router
to extend a circuit one hop
avg_rank(self)
bw_ratio(self)
Return the ratio of the Router's advertised bandwidth to its 
observed average stream bandwidth
bw_ratio_ratio(self)
circ_fail_rate(self)
circ_fail_ratio(self)
circ_suspect_rate(self)
circ_suspect_ratio(self)
current_uptime(self)
failed_per_hour(self)
Return the number of circuit extend failures per hour for this 
Router
reset(self)
Reset all stats on this Router
sanity_check(self)
Makes sure all stats are self-consistent
strm_bw_ratio(self)
Return the ratio of the Router's stream capacity to the average
stream capacity passed in as 'mean'
strm_fail_rate(self)
strm_fail_ratio(self)
strm_suspect_rate(self)
strm_suspect_ratio(self)
suspected_per_hour(self)
Return the number of circuits that failed with this router as an
earlier hop
was_used(self)
Return True if this router was used in this round

Data and other attributes defined here:
global_bw_mean = 0.0
global_cf_mean = 0.0
global_cs_mean = 0.0
global_ratio_dev = 0.0
global_ratio_mean = 0.0
global_sf_mean = 0.0
global_ss_mean = 0.0
global_strm_dev = 0.0
global_strm_mean = 0.0
key = 'Metatroller Router Statistics:\n CC=Circuits Cho...-ratio) SR=Global mean/mean BW U=Uptime (h)\n'

Methods inherited from TorCtl.TorCtl.Router:
update_to(self, new)
Somewhat hackish method to update this router to be a copy of
'new'
will_exit_to(self, ip, port)
Check the entire exitpolicy to see if the router will allow
connections to 'ip':'port'

Data and other attributes inherited from TorCtl.TorCtl.Router:
build_from_desc = <TorCtl.TorUtil.Callable instance at 0x82634cc>

 
class SuspectRouterList(ReasonRouterList)
    Helper class to track all routers suspected of failing for a given
reason. The main difference between this and the normal
ReasonRouterList is the sort order and the verification.
 
  Methods defined here:
__init__(self, reason)
sort_list(self)

Methods inherited from ReasonRouterList:
add_r(self, r)
Add a router to the list for this reason
total_failed(self)
Get a list of total failures for this reason
total_suspected(self)
Get a list of total suspected failures for this reason
write_list(self, f)
Write the list of failure counts for this reason 'f'

 
Data
        control_host = '127.0.0.1'
control_port = 9061
logfile = None
loglevel = 'DEBUG'
meta_host = '127.0.0.1'
meta_port = 9052