Repository navigation
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
Function Fitting Utilities
including
Social Activity Pulse
The surge or pulse of social media activity can be characterized by the time it takes to reach
its peak, the height of the peak, the 1/2 life of the surge in activity, etc. To
compare stories or media souces, it is useful to fit the data to a standard pulse function in
order to compare attributes consistently.
See https://gnip.com.s3.amazonaws.com/ScottHendrickson/SocialMediaPulse.pdf for additional information.
GAMMA DISTRIBUTION
==================
When the event is characterized by a homogenious poisson counting process, the social media pulse
is modeled by a gamma distribution.
A(t) = beta^-alpha * A0*(t-t0)^(alpha-1) * exp(-A0*(t-t0)/beta) / gamma(a)
t - time
t0 - start of pulse
alpha, beta are parameters describing magnitude, geometric growth and exponential decay.
A0 - normalize the values to estimate probability, useful for fitting counts directly
To ensure numerical stability, it is helpful to scale t. A simple utility scale.py is included.
Scale.py adds a column immediately following the t column you specify that is normalized from
z = (t-tmin)/(tmax - tmin).
DOUBLE EXPONENTIAL
==================
One convenient model is that the traffic follows a double exponential function. (This may
be useful as a slightly more analytially tractable estimate to the gamma distirbution.)
Example: See the graph at http://blog.gnip.com/wp-content/uploads/2012/01/pubfits.png (associated
with the article at http://blog.gnip.com/twitter-shouts-huntsmans-out/)
The fit.py utility fits the pulse in activity to a double exponential model of the pulse:
A(t) = A0 * (1 -exp(-alpha * (t-t0))) * exp(-beta * (t-t0))
t - time
t0 - start of pulse
A0, alpha, beta are parameters describing magnitude, growth and decay.
Two versions of exponential fits are included as well (they accomplish the same fits, but
have different numerical stability attributes.)
FITS
====
fit.py requires the scipy module.
Try to window the data to the pulse before fitting.
Usage:
Example 1:
To fit the function, send data to standard input (fit.py).
This will calculate the best fit for A0 (shown as A below), alpha, beta, the starting time
of the pulse (t0 = tOffset), the time at the peak, the value at the peak, the average (first
moment) of the the pulse and the function value at tavg, the area under the curve (Norm)
and the 1/2 life-with the associated function value at t1/2.
If you don't specify any arguments, fit.py assumes your data consists of 2 columns (text headers ok)
and doubleexp_function.py tries to guess initial parameters for the fit.
For instance, using the example file provided,
> cat pulse.csv | ./fit.py
# A=30056.640395
# alpha=706.789547
# beta=30.533273
# tOffset=0.000099
# TPeak=0.004604, FPeak=25109.169342
# TAvgT=0.034206, FAvg=10608.680897
# Norm=943.625186
# T1/2life=0.028691, F1/2Life=12554.584671
0.0,0.0,0.0,0.0
0.0010101010101,13878.536414,0.0010101010101,14480.8439804
0.0020202020202,21053.7409876,0.0020202020202,19463.6955929
0.0030303030303,24021.5650303,0.0030303030303,23247.1318939
0.0040404040404,25004.8839646,0.0040404040404,27212.7301957
0.00505050505051,25058.8100021,0.00505050505051,26272.1642341
0.00606060606061,24683.9631792,0.00606060606061,25636.5140401
0.00707070707071,24117.6785968,0.00707070707071,24259.3154772
0.00808080808081,23472.2838504,0.00808080808081,25528.2671864
0.00909090909091,22800.7587834,0.00909090909091,20681.9249606
0.010101010101,22127.9132373,0.010101010101,22121.6544239
0.0111111111111,21465.1912655,0.0111111111111,19854.0500165
0.0131313131313,20187.5496541,0.0131313131313,19291.9769569
0.0141414141414,19575.43309,0.0141414141414,19707.6030213
0.0161616161616,18405.125556,0.0161616161616,17683.251643
0.0171717171717,17846.2515313,0.0171717171717,16736.2283576
0.0191919191919,16778.7713627,0.0191919191919,16471.1023042
0.020202020202,16269.195805,0.020202020202,17305.3271536
0.0222222222222,15295.9888315,0.0222222222222,14703.1360801
0.0232323232323,14831.4365503,0.0232323232323,15589.5707733
...
NOTE: doubleexp_function.py now outputs an additional time column at position 3 based on the t0 = 0.
This is useful for giving pulses a common baseline for comparing half-lives, peaks, etc
on a graph.
In addition to the fit parameters, the output also provides the function evaluated at the
input data (time) points (first 2 columns) along side the original data (remaining columns).
This makes it simple to calculate error or visualize the pulse against the data.
Example 2:
Fit to gamma distribution,
> cat pulse.csv | ./fit.py -fgamma
# x0=0.002020
# a=1.026359
# b=0.031917
# A0=919.468975
# tOffset=0.002020
# tavg=0.034779 f(tavg)=10481.683577i
# tpeak=0.002861 f(tpeak)=25870.541585
0.0,0.0,0.0,0.0
0.0010101010101,14480.8439804,0.0010101010101,0.0
0.0020202020202,19463.6955929,0.0020202020202,19463.6851941
0.0030303030303,23247.1318939,0.0030303030303,25858.4136042
0.0040404040404,27212.7301957,0.0040404040404,25514.8009723
0.00505050505051,26272.1642341,0.00505050505051,24985.5731713
0.00606060606061,25636.5140401,0.00606060606061,24391.4798319
0.00707070707071,24259.3154772,0.00707070707071,23771.0403792
0.00808080808081,25528.2671864,0.00808080808081,23141.4691525
0.00909090909091,20681.9249606,0.00909090909091,22511.8498162
0.010101010101,22121.6544239,0.010101010101,21887.4618773
0.0111111111111,19854.0500165,0.0111111111111,21271.5602281
0.0131313131313,19291.9769569,0.0131313131313,20072.7914575
...
Example 3:
To specify initial values of fit parameters, use,
fit.py -p"[A, alpha, beta, offset]"
Example 4:
To specify an alternate column for fit data, use -c (counting starts at 0). For example, if time is in
column 1 and your data is in column 4:
fit.py -c3
Example 5:
To specify a custom function evaluation range, use -r.
fit.py -r[0,10,100]
Outputs an outer join of the data and the fit function evaluated at the points in the range specified,
in this case 100 points spanning 0 to 10.
Example 6:
Getting help:
> ./fit.py -h
Usage: fit.py [options]
Options:
-h, --help show this help message and exit
-p INIT_PARAMETERS, --init-parameters=INIT_PARAMETERS
Iinitial guess of parameters [A, alpha, beta, offset]
([] or blank ok)
-i ICOLUMN, --column-independent=ICOLUMN
column of independent variable to fit (default is 1)
-c COLUMN, --column-dependent=COLUMN
Column of dependent variable to fit (default is 2)
-l LABEL, --append-label=LABEL
Append label column (useful as R factor)
-f FUNC_NAME, --func-name=FUNC_NAME
gamma - gamma distribution; exp - exponential; dubex -
Pulse Function (default); lognorm - log-normal CDF;
gauss - gaussian PDF
-r RANGE_STRING, --range-list=RANGE_STRING
Function evaluation output range as '[start time, end
time, number of points]' If not set, evaluate at fit
points.
(Or read the code.)
For more information, contact shendrickson@gnip.com. Social Media Pulse by Scott Hendrickson is licensed
under a Creative Commons Attribution-ShareAlike 3.0 Unported License
(see http://creativecommons.org/licenses/by-sa/3.0/)