gensim/docs/dist_lsi.html at ESA · blehman/gensim

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<head>

<title>Distributed Latent Semantic Analysis — gensim</title>

var DOCUMENTATION_OPTIONS = {

URL_ROOT: '',

VERSION: '0.8.6',

COLLAPSE_INDEX: false,

FILE_SUFFIX: '.html',

HAS_SOURCE: true

};

</script>

<!-- twitter search widget

-->

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-24066335-1']);

_gaq.push(['_trackPageview']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

})();

</script>

</head>

<body>

<h3>Navigation</h3>

<ul>

<a href="genindex.html" title="General Index"

accesskey="I">index</a></li>

<a href="py-modindex.html" title="Python Module Index"

>modules</a> |</li>

<a href="dist_lda.html" title="Distributed Latent Dirichlet Allocation"

accesskey="N">next</a> |</li>

<a href="distributed.html" title="Distributed Computing"

accesskey="P">previous</a> |</li>

<li><a href="index.html">Gensim home</a>| </li>

<li><a href="tutorial.html">Tutorials</a>| </li>

<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>

<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>

<li><a href="apiref.html">API reference</a>»</li>

<li><a href="distributed.html" accesskey="U">Distributed Computing</a> »</li>

</ul>

</div>

<h3><a href="index.html">Table Of Contents</a></h3>

<ul>

<li><a class="reference internal" href="#">Distributed Latent Semantic Analysis</a><ul>

<li><a class="reference internal" href="#setting-up-the-cluster">Setting up the cluster</a></li>

<li><a class="reference internal" href="#running-lsa">Running LSA</a><ul>

<li><a class="reference internal" href="#distributed-lsa-on-wikipedia">Distributed LSA on Wikipedia</a></li>

</ul>

</li>

</ul>

</li>

</ul>

<h4>Previous topic</h4>

<a href="distributed.html"

title="previous chapter">Distributed Computing</a>

<h4>Next topic</h4>

<a href="dist_lda.html"

title="next chapter">Distributed Latent Dirichlet Allocation</a>

<h3>Quick search</h3>

</form>

Enter search terms or a module, class or function name.

</div>

</div>

<h1>Distributed Latent Semantic Analysis<a class="headerlink" href="#distributed-latent-semantic-analysis" title="Permalink to this headline">¶</a></h1>

Note

See <a class="reference internal" href="distributed.html">Distributed Computing</a> for an introduction to distributed computing in <cite>gensim</cite>.

</div>

<h2>Setting up the cluster<a class="headerlink" href="#setting-up-the-cluster" title="Permalink to this headline">¶</a></h2>

We will show how to run distributed Latent Semantic Analysis by means of an example.

Let’s say we have 5 computers at our disposal, all on the same network segment (=reachable

by network broadcast). To start with, install <cite>gensim</cite> and <cite>Pyro</cite> on each computer with:

<div class="highlight-python"><pre>$ sudo easy_install gensim[distributed]</pre>

</div>

and run Pyro’s name server on exactly one of the machines (doesn’t matter which one):

<div class="highlight-python"><pre>$ python -m Pyro4.naming -n 0.0.0.0 &</pre>

</div>

Let’s say our example cluster consists of dual-core computers with loads of

memory. We will therefore run two worker scripts on four of the physical machines,

creating eight logical worker nodes:

<div class="highlight-python"><pre>$ python -m gensim.models.lsi_worker &</pre>

</div>

This will execute <cite>gensim</cite>‘s <cite>lsi_worker.py</cite> script (to be run twice on each of the

four computer).

This lets <cite>gensim</cite> know that it can run two jobs on each of the four computers in

parallel, so that the computation will be done faster, while also taking up twice

as much memory on each machine.

Next, pick one computer that will be a job scheduler in charge of worker

synchronization, and on it, run <cite>LSA dispatcher</cite>. In our example, we will use the

fifth computer to act as the dispatcher and from there run:

<div class="highlight-python"><pre>$ python -m gensim.models.lsi_dispatcher &</pre>

</div>

In general, the dispatcher can be run on the same machine as one of the worker nodes, or it

can be another, distinct computer (within the same broadcast domain). The dispatcher

won’t be doing much with CPU most of the time, but pick a computer with ample memory.

And that’s it! The cluster is set up and running, ready to accept jobs. To remove

a worker later on, simply terminate its <cite>lsi_worker</cite> process. To add another worker, run another

<cite>lsi_worker</cite> (this will not affect a computation that is already running, the additions/deletions are not dynamic).

If you terminate <cite>lsi_dispatcher</cite>, you won’t be able to run computations until you run it again

(surviving worker processes can be re-used though).

</div>

<h2>Running LSA<a class="headerlink" href="#running-lsa" title="Permalink to this headline">¶</a></h2>

So let’s test our setup and run one computation of distributed LSA. Open a Python

shell on one of the five machines (again, this can be done on any computer

in the same <a class="reference external" href="http://en.wikipedia.org/wiki/Broadcast_domain">broadcast domain</a>,

our choice is incidental) and try:

<div class="highlight-python"><div class="highlight"><pre>>>> from gensim import corpora, models, utils

>>> import logging

>>> logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

>>> corpus = corpora.MmCorpus('/tmp/deerwester.mm') # load a corpus of nine documents, from the Tutorials

>>> id2word = corpora.Dictionary.load('/tmp/deerwester.dict')

>>> lsi = models.LsiModel(corpus, id2word=id2word, num_topics=200, chunksize=1, distributed=True) # run distributed LSA on nine documents

</pre></div>

</div>

This uses the corpus and feature-token mapping created in the <a class="reference internal" href="tut1.html">Corpora and Vector Spaces</a> tutorial.

If you look at the log in your Python session, you should see a line similar to:

<div class="highlight-python"><pre>2010-08-09 23:44:25,746 : INFO : using distributed version with 8 workers</pre>

</div>

which means all went well. You can also check the logs coming from your worker and dispatcher

processes — this is especially helpful in case of problems.

To check the LSA results, let’s print the first two latent topics:

<div class="highlight-python"><div class="highlight"><pre>>>> lsi.print_topics(num_topics=2, num_words=5)

topic #0(3.341): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response"

topic #1(2.542): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system"

</pre></div>

</div>

Success! But a corpus of nine documents is no challenge for our powerful cluster...

In fact, we had to lower the job size (<cite>chunksize</cite> parameter above) to a single document

at a time, otherwise all documents would be processed by a single worker all at once.

So let’s run LSA on one million documents instead:

<div class="highlight-python"><div class="highlight"><pre>>>> # inflate the corpus to 1M documents, by repeating its documents over&over

>>> corpus1m = utils.RepeatCorpus(corpus, 1000000)

>>> # run distributed LSA on 1 million documents

>>> lsi1m = models.LsiModel(corpus1m, id2word=id2word, num_topics=200, chunksize=10000, distributed=True)

>>> lsi1m</s C6A9 pan>.print_topics(num_topics=2, num_words=5)

topic #0(1113.628): 0.644*"system" + 0.404*"user" + 0.301*"eps" + 0.265*"time" + 0.265*"response"

topic #1(847.233): 0.623*"graph" + 0.490*"trees" + 0.451*"minors" + 0.274*"survey" + -0.167*"system"

</pre></div>

</div>

The log from 1M LSA should look like:

<div class="highlight-python"><pre>2010-08-10 02:46:35,087 : INFO : using distributed version with 8 workers

2010-08-10 02:46:35,087 : INFO : updating SVD with new documents

2010-08-10 02:46:35,202 : INFO : dispatched documents up to #10000

2010-08-10 02:46:35,296 : INFO : dispatched documents up to #20000

...

2010-08-10 02:46:46,524 : INFO : dispatched documents up to #990000

2010-08-10 02:46:46,694 : INFO : dispatched documents up to #1000000

2010-08-10 02:46:46,694 : INFO : reached the end of input; now waiting for all remaining jobs to finish

2010-08-10 02:46:47,195 : INFO : all jobs finished, downloading final projection

2010-08-10 02:46:47,200 : INFO : decomposition complete</pre>

</div>

Due to the small vocabulary size and trivial structure of our “one-million corpus”, the computation

of LSA still takes only 12 seconds. To really stress-test our cluster, let’s do

Latent Semantic Analysis on the English Wikipedia.

<h3>Distributed LSA on Wikipedia<a class="headerlink" href="#distributed-lsa-on-wikipedia" title="Permalink to this headline">¶</a></h3>

First, download and prepare the Wikipedia corpus as per <a class="reference internal" href="wiki.html">Experiments on the English Wikipedia</a>, then load

the corpus iterator with:

<div class="highlight-python"><div class="highlight"><pre>>>> import logging, gensim, bz2

>>> # load id->word mapping (the dictionary)

>>> id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')

>>> # load corpus iterator

>>> mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')

>>> # mm = gensim.corpora.MmCorpus(bz2.BZ2File('wiki_en_tfidf.mm.bz2')) # use this if you compressed the TFIDF output

>>> print mm

MmCorpus(3199665 documents, 100000 features, 495547400 non-zero entries)

</pre></div>

</div>

Now we’re ready to run distributed LSA on the English Wikipedia:

<div class="highlight-python"><div class="highlight"><pre>>>> # extract 400 LSI topics, using a cluster of nodes

>>> lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400, chunksize=20000, distributed=True)

>>> # print the most contributing words (both positively and negatively) for each of the first ten topics

>>> lsi.print_topics(10)

2010-11-03 16:08:27,602 : INFO : topic #0(200.990): -0.475*"delete" + -0.383*"deletion" + -0.275*"debate" + -0.223*"comments" + -0.220*"edits" + -0.213*"modify" + -0.208*"appropriate" + -0.194*"subsequent" + -0.155*"wp" + -0.117*"notability"

2010-11-03 16:08:27,626 : INFO : topic #1(143.129): -0.320*"diff" + -0.305*"link" + -0.199*"image" + -0.171*"www" + -0.162*"user" + 0.149*"delete" + -0.147*"undo" + -0.144*"contribs" + -0.122*"album" + 0.113*"deletion"

2010-11-03 16:08:27,651 : INFO : topic #2(135.665): -0.437*"diff" + -0.400*"link" + -0.202*"undo" + -0.192*"user" + -0.182*"www" + -0.176*"contribs" + 0.168*"image" + -0.109*"added" + 0.106*"album" + 0.097*"copyright"

2010-11-03 16:08:27,677 : INFO : topic #3(125.027): -0.354*"image" + 0.239*"age" + 0.218*"median" + -0.213*"copyright" + 0.204*"population" + -0.195*"fair" + 0.195*"income" + 0.167*"census" + 0.165*"km" + 0.162*"households"

2010-11-03 16:08:27,701 : INFO : topic #4(116.927): -0.307*"image" + 0.195*"players" + -0.184*"median" + -0.184*"copyright" + -0.181*"age" + -0.167*"fair" + -0.162*"income" + -0.151*"population" + -0.136*"households" + -0.134*"census"

2010-11-03 16:08:27,728 : INFO : topic #5(100.326): 0.501*"players" + 0.318*"football" + 0.284*"league" + 0.193*"footballers" + 0.142*"image" + 0.133*"season" + 0.119*"cup" + 0.113*"club" + 0.110*"baseball" + 0.103*"f"

2010-11-03 16:08:27,754 : INFO : topic #6(92.298): -0.411*"album" + -0.275*"albums" + -0.217*"band" + -0.214*"song" + -0.184*"chart" + -0.163*"songs" + -0.160*"singles" + -0.149*"vocals" + -0.139*"guitar" + -0.129*"track"

2010-11-03 16:08:27,780 : INFO : topic #7(83.811): -0.248*"wikipedia" + -0.182*"keep" + 0.180*"delete" + -0.167*"articles" + -0.152*"your" + -0.150*"my" + 0.144*"film" + -0.130*"we" + -0.123*"think" + -0.120*"user"

2010-11-03 16:08:27,807 : INFO : topic #8(78.981): 0.588*"film" + 0.460*"films" + -0.130*"album" + -0.127*"station" + 0.121*"television" + 0.115*"poster" + 0.112*"directed" + 0.110*"actors" + -0.096*"railway" + 0.086*"movie"

2010-11-03 16:08:27,834 : INFO : topic #9(78.620): 0.502*"kategori" + 0.282*"categoria" + 0.248*"kategorija" + 0.234*"kategorie" + 0.172*"категория" + 0.165*"categoría" + 0.161*"kategoria" + 0.148*"categorie" + 0.126*"kategória" + 0.121*"catégorie"

</pre></div>

</div>

In serial mode, creating the LSI model of Wikipedia with this one-pass algorithm

takes about 5.25h on my laptop (OS X, C2D 2.53GHz, 4GB RAM with <cite>libVec</cite>).

In distributed mode with four workers (Linux, dual-core Xeons of 2Ghz, 4GB RAM

with <cite>ATLAS</cite>), the wallclock time taken drops to 1 hour and 41 minutes. You can

read more about various internal settings and experiments in my <a class="reference external" href="http://nlp.fi.muni.cz/~xrehurek/nips/rehurek_nips.pdf">research

paper</a>.

</div>

</div>

<h3>Navigation</h3>

<ul>

<a href="genindex.html" title="General Index"

>index</a></li>

<a href="py-modindex.html" title="Python Module Index"

>modules</a> |</li>

<a href="dist_lda.html" title="Distributed Latent Dirichlet Allocation"

>next</a> |</li>

<a href="distributed.html" title="Distributed Computing"

>previous</a> |</li>

<li><a href="index.html">Gensim home</a>| </li>

<li><a href="tutorial.html">Tutorials</a>| </li>

<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>

<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>

<li><a href="apiref.html">API reference</a>»</li>

<li><a href="distributed.html" >Distributed Computing</a> »</li>

</ul>

</div>

Last updated on Sep 16, 2012.

</div>

</body>

</html>

FilesExpand file tree

dist_lsi.html

Latest commit

History

dist_lsi.html

File metadata and controls