Skip to main content

用于分析 GMANE 电子邮件列表数据库的实用程序

项目描述

该项目提供了用于分析 GMANE 电子邮件数据库的辅助类。安装:

$ pip install gmaneLegacy

或者

$ python setup.py 安装

为了更好地控制自定义(和调试),克隆 repo 并使用带有 -e 的 pip 安装:

$ git clone https://github.com/ttm/gmaneLegacy.git

$ pip install -e <path_to_repo>

此安装方法对于来自 IPython.lib.deepreload 和标准 importlib 的重新加载功能特别有用。

功能基于有关交互网络的物理文章:[1] 人类交互网络的稳定性:顶点的原始类型、度量的突出性和活动统计:http ://arxiv.org/abs/1310.7769 [2] 文本的连接区分交互网络中的生产:http: //arxiv.org/abs/1412.7309 [3] Versinus:进化中图形的可视化方法:http ://arxiv.org/abs/1412.7311

具有1)拓扑结构分析的核心概念;2)文本制作分析;3) 演化结构的可视化。活动的时间分布和参与者之间的分布也通过特定的例程和间接通过 1)、2) 和 3) 进行。

理想情况下,这个包应该简化: - 下载 GMNE 电子邮件列表数据。- 使用下载的数据构建基本数据结构。- 通过复杂的网络和 NLP 标准分析数据。- 通过多种布局方法进行可视化。

PS。根据 [1] 在网络代理活动中手动实施对称测量(在网络和数字包中未找到)。PS2。在 tests/newTextTables.py 和 tests/makeOverallTextAnalysis.py PS 中正在进行的研究。还要检查 gmane Python 包https://github.com/ttm/gmane

使用示例

从一个 GMane 列表下载消息:

import gmane as g
dl=g.DownloadGmaneData() # saves into ~/.gmane/
dl.downloadListsIDS() # acquires all GMANE list_ids
dl.downloadListMessages(dl.list_ids[100])
dl.cleanDownloadedLists() # remove empty messages for coherence
dl.downloadedStats() # creates ~/.gmane/stats.txt

# to load message contents to Python objects:
# load 10 messages from list with list_id gmane.ietf.rfc822
lm=g.LoadMessages("gmane.ietf.rfc822",10)

# or access the structures downloaded to your filesystem
dl=g.DownloadGmaneData()
dl.getDownloadedLists()
lms=[]
# and download all messages from 5 lists
for list_id in dl.downloaded_lists[:5]:
    lms.append(g.LoadMessages(list_id))

# to load first three lists with the greated number
# of downloaded messages:
dl.downloadedStats() # might take a while
load_msgs=[]
for list_stat in dl.lists[:3]:
    list_id=list_stat[0]
    load_msgs.append(g.LoadMessages(list_id))

# to make basic datastructures of a list with
# greatest number of messages:
ds=g.MessageDataStructures(load_msgs[0])
mm=ds.messages
ids=ds.message_ids
print("first: ", mm[ids[0]][2], "last:", mm[ids[-1]][2])

# circular (directional) statistics for activity along time
# (hours of the day, days of the week, days of the month, etc):
# mean_vec, mean_angle, size_mean_vec, circular_mean,
# circular_variance, circular dispersion
# and histograms
ts=g.TimeStatistics(ds)
print("made overall circular activity statistics along time")

# make latex tables to observe distributions within bins of interest
hi=100*ts.hours["histogram"]/ts.hours["histogram"].sum()
row_labels=list(range(24))
tstring=g.parcialSums(row_labels,data=[hi],partials=[1,2,3,4,6,12],
            partial_labels=["h","2h","3h","4h","6h","12h"],datarow_labels=["APACHE"])
g.writeTex(tstring,"here.tex")

ps=g.AgentStatistics(ds)
print("made overall activity statistics among participants")

# build the interaction network of the messages:
nw=g.InteractionNetwok(ds)

print("number of nodes: {}, number of edges: {}".format(
nw.g.number_of_nodes(), nw.g.number_of_edges()))

nm=g.NetworkMeasures(nw) # take measures, including symmetry related measures
np=g.NetworkPartitioning(nm) # partition in primitive typology
sa=np.sectorialized_agents # get members of each sector
print("{} agents in periphery, {} are intermediary and {} hubs".format(sa[0],sa[1],sa[2]))
sa=np.sectorialized_agents__ # smoothed histogram for classification
print("{} agents in periphery, {} are intermediary and {} hubs".format(sa[0],sa[1],sa[2]))

# draw
nd=g.NetworkDrawer()
print("drawer started")
nd.makeLayout(nm)
print("gave (x,y) for each author with 5-15-80")
nd2=g.NetworkDrawer()
print("drawer two started")
nd2.makeLayout(nm,np)
print("gave (x,y) for each author with \
sectors by comparison with Erdos-Renyi")
nd.drawNetwork( iN,nm ,"test.png")
nd2.drawNetwork( iN,nm,"test2.png")

# make basic PCA plots of network measures:
npca=g.NetworkPCA(nm)
# Plot PCA with a colored primitive sectors
npca=g.NetworkPCA(nm,np)

# Evolves network with measures, partitions,
# PCA, principal components and Versinus plots saved to disk
lm=lms[0] # loaded messages from list with most messages
ne=g.NetworkEvolution(step_size=10)
ne.evolveRaw(lm.messages,imagerate=4,erdos_sectors=True)
# ne.makeVideo() use this to avoid evolving again just to make video
# see testDrawer.py or g.NetworkEvolution to make movies:
# https://www.youtube.com/watch?v=iS8NwEy291g

# after making network evolution measurements and video,
# you can both make music:
em=g.EvolutionMusic()
print("music is done")
# avconv -i mixY.wav -i evo[..<depends on the evolution done>..].avi final.avi
# delivers you the final.avi animation with a soundtrack relative to network measures
# currently it is the 'four hubs dance' by default:
# https://www.youtube.com/watch?v=YxDiwzAUPeU

# and further analysis of measures and Erdos sectors:
et=g.EvolutionTimelines()
print("Written png files with network measures along evolution timeline")

# Enjoy!

进一步的文档在测试/文件夹和对象文档字符串中。

下载文件

下载适用于您平台的文件。如果您不确定要选择哪个,请了解有关安装包的更多信息。

源分布

gmaneLegacy-0.1.dev27.tar.gz (56.4 kB 查看哈希

已上传 source