Repository navigation
Expand file tree
/
Copy pathsimserver.html
More file actions
462 lines (431 loc) · 36 KB
/
Copy pathsimserver.html
File metadata and controls
462 lines (431 loc) · 36 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Document Similarity Server — gensim</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '',
VERSION: '0.8.6',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="author" title="About these documents" href="about.html" />
<link rel="top" title="gensim" href="index.html" />
<!-- twitter search widget
<script type="text/javascript" src="_static/widget.js"></script>
-->
<meta property="og:title" content="#gensim" />
<meta property="og:description" content="Efficient topic modelling in Python" />
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-24066335-1']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Document Similarity Server</a><ul>
<li><a class="reference internal" href="#what-is-a-document-similarity-service">What is a document similarity service?</a></li>
<li><a class="reference internal" href="#what-is-it-good-for">What is it good for?</a></li>
<li><a class="reference internal" href="#how-is-it-unique">How is it unique?</a></li>
<li><a class="reference internal" href="#prerequisites">Prerequisites</a></li>
<li><a class="reference internal" href="#what-is-a-document">What is a document?</a></li>
<li><a class="reference internal" href="#what-is-a-corpus">What is a corpus?</a></li>
<li><a class="reference internal" href="#wait-upload-what-where">Wait, upload what, where?</a></li>
<li><a class="reference internal" href="#model-training">Model training</a></li>
<li><a class="reference internal" href="#indexing-documents">Indexing documents</a></li>
<li><a class="reference internal" href="#querying">Querying</a></li>
<li><a class="reference internal" href="#remote-access">Remote access</a></li>
<li><a class="reference internal" href="#concurrency">Concurrency</a></li>
<li><a class="reference internal" href="#what-do-you-mean-visible">What do you mean, visible?</a></li>
<li><a class="reference internal" href="#other-stuff">Other stuff</a></li>
</ul>
</li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" size="24" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="document-similarity-server">
<span id="simserver"></span><h1>Document Similarity Server<a class="headerlink" href="#document-similarity-server" title="Permalink to this headline">¶</a></h1>
<p>The 0.7.x series of <a class="reference external" href="http://radimrehurek.com/gensim/">gensim</a> was about improving performance and consolidating API.
0.8.x will be about new features — 0.8.1, first of the series, is a <strong>document similarity service</strong>.</p>
<p>The source code itself has been moved from gensim to its own, dedicated package, named <cite>simserver</cite>.
Get it from <a class="reference external" href="http://pypi.python.org/pypi/simserver">PyPI</a> or clone it on <a class="reference external" href="https://github.com/piskvorky/gensim-simserver">Github</a>.</p>
<div class="section" id="what-is-a-document-similarity-service">
<h2>What is a document similarity service?<a class="headerlink" href="#what-is-a-document-similarity-service" title="Permalink to this headline">¶</a></h2>
<p>Conceptually, a service that lets you :</p>
<ol class="arabic simple">
<li>train a semantic model from a corpus of plain texts (no manual annotation and mark-up needed)</li>
<li>index arbitrary documents using this semantic model</li>
<li>query the index for similar documents (the query can be either an id of a document already in the index, or an arbitrary text)</li>
</ol>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">simserver</span> <span class="kn">import</span> <span class="n">SessionServer</span>
<span class="gp">>>> </span><span class="n">server</span> <span class="o">=</span> <span class="n">SessionServer</span><span class="p">(</span><span class="s">'/tmp/my_server'</span><span class="p">)</span> <span class="c"># resume server (or create a new one)</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">training_corpus</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s">'lsi'</span><span class="p">)</span> <span class="c"># create a semantic model</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">some_documents</span><span class="p">)</span> <span class="c"># convert plain text to semantic representation and index it</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="n">query</span><span class="p">)</span> <span class="c"># convert query to semantic representation and compare against index</span>
<span class="gp">>>> </span><span class="o">...</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">more_documents</span><span class="p">)</span> <span class="c"># add to index: incremental indexing works</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="gp">>>> </span><span class="o">...</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">ids_to_delete</span><span class="p">)</span> <span class="c"># incremental deleting also works</span>
<span class="gp">>>> </span><span class="n">server</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="gp">>>> </span><span class="o">...</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">“Semantic” here refers to semantics of the crude, statistical type –
<a class="reference external" href="http://en.wikipedia.org/wiki/Latent_semantic_analysis">Latent Semantic Analysis</a>,
<a class="reference external" href="http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation">Latent Dirichlet Allocation</a> etc.
Nothing to do with the semantic web, manual resource tagging or detailed linguistic inference.</p>
</div>
</div>
<div class="section" id="what-is-it-good-for">
<h2>What is it good for?<a class="headerlink" href="#what-is-it-good-for" title="Permalink to this headline">¶</a></h2>
<p>Digital libraries of (mostly) text documents. More generally, it helps you annotate,
organize and navigate documents in a more abstract way, compared to plain keyword search.</p>
</div>
<div class="section" id="how-is-it-unique">
<h2>How is it unique?<a class="headerlink" href="#how-is-it-unique" title="Permalink to this headline">¶</a></h2>
<ol class="arabic simple">
<li><strong>Memory independent</strong>. Gensim has unique algorithms for statistical analysis that allow
you to create semantic models of arbitrarily large training corpora (larger than RAM) very quickly
and in constant RAM.</li>
<li><strong>Memory independent (again)</strong>. Indexing shards are stored as files to disk/mmapped back as needed,
so you can index very large corpora. So again, constant RAM, this time independent of the number of indexed documents.</li>
<li><strong>Efficient</strong>. Gensim makes heavy use of Python’s NumPy and SciPy libraries to make indexing and
querying efficient.</li>
<li><strong>Robust</strong>. Modifications of the index are transactional, so you can commit/rollback an
entire indexing session. Also, during the session, the service is still available
for querying (using its state from when the session started). Power failures leave
service in a consistent state (implicit rollback).</li>
<li><strong>Pure Python</strong>. Well, technically, NumPy and SciPy are mostly wrapped C and Fortran, but
<a class="reference external" href="http://radimrehurek.com/gensim/">gensim</a> itself is pure Python. No compiling, installing or root priviledges needed.</li>
<li><strong>Concurrency support</strong>. The underlying service object is thread-safe and can
therefore be used as a daemon server: clients connect to it via RPC and issue train/index/query requests remotely.</li>
<li><strong>Cross-network, cross-platform and cross-language</strong>. While the Python server runs
over TCP using <a class="reference external" href="http://irmen.home.xs4all.nl/pyro/">Pyro</a>,
clients in Java/.NET are trivial thanks to <a class="reference external" href="http://irmen.home.xs4all.nl/pyrolite/">Pyrolite</a>.</li>
</ol>
<p>The rest of this document serves as a tutorial explaining the features in more detail.</p>
</div>
<hr class="docutils" />
<div class="section" id="prerequisites">
<h2>Prerequisites<a class="headerlink" href="#prerequisites" title="Permalink to this headline">¶</a></h2>
<p>It is assumed you have <cite>gensim</cite> properly <a class="reference internal" href="install.html"><em>installed</em></a>. You’ll also
need the <a class="reference external" href="http://pypi.python.org/pypi/sqlitedict">sqlitedict</a> package that wraps
Python’s sqlite3 module in a thread-safe manner:</p>
<div class="highlight-python"><pre>$ sudo easy_install -U sqlitedict</pre>
</div>
<p>To test the remote server capabilities, install Pyro4 (Python Remote Objects, at
version 4.8 as of this writing):</p>
<div class="highlight-python"><pre>$ sudo easy_install Pyro4</pre>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p>Don’t forget to initialize logging to see logging messages:</p>
<div class="last highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">logging</span>
<span class="gp
8000
">>>> </span><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(asctime)s</span><span class="s"> : </span><span class="si">%(levelname)s</span><span class="s"> : </span><span class="si">%(message)s</span><span class="s">'</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="what-is-a-document">
<h2>What is a document?<a class="headerlink" href="#what-is-a-document" title="Permalink to this headline">¶</a></h2>
<p>In case of text documents, the service expects:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">document</span> <span class="o">=</span> <span class="p">{</span><span class="s">'id'</span><span class="p">:</span> <span class="s">'some_unique_string'</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">'tokens'</span><span class="p">:</span> <span class="p">[</span><span class="s">'content'</span><span class="p">,</span> <span class="s">'of'</span><span class="p">,</span> <span class="s">'the'</span><span class="p">,</span> <span class="s">'document'</span><span class="p">,</span> <span class="s">'...'</span><span class="p">],</span>
<span class="gp">>>> </span> <span class="s">'other_fields_are_allowed_but_ignored'</span><span class="p">:</span> <span class="bp">None</span><span class="p">}</span>
</pre></div>
</div>
<p>This format was chosen because it coincides with plain JSON and is therefore easy to serialize and send over the wire, in almost any language.
All strings involved must be utf8-encoded.</p>
</div>
<div class="section" id="what-is-a-corpus">
<h2>What is a corpus?<a class="headerlink" href="#what-is-a-corpus" title="Permalink to this headline">¶</a></h2>
<p>A sequence of documents. Anything that supports the <cite>for document in corpus: ...</cite>
iterator protocol. Generators are ok. Plain lists are also ok (but consume more memory).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">gensim</span> <span class="kn">import</span> <span class="n">utils</span>
<span class="gp">>>> </span><span class="n">texts</span> <span class="o">=</span> <span class="p">[</span><span class="s">"Human machine interface for lab abc computer applications"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"A survey of user opinion of computer system response time"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"The EPS user interface management system"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"System and human system engineering testing of EPS"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"Relation of user perceived response time to error measurement"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"The generation of random binary unordered trees"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"The intersection graph of paths in trees"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"Graph minors IV Widths of trees and well quasi ordering"</span><span class="p">,</span>
<span class="gp">>>> </span> <span class="s">"Graph minors A survey"</span><span class="p">]</span>
<span class="gp">>>> </span><span class="n">corpus</span> <span class="o">=</span> <span class="p">[{</span><span class="s">'id'</span><span class="p">:</span> <span class="s">'doc_</span><span class="si">%i</span><span class="s">'</span> <span class="o">%</span> <span class="n">num</span><span class="p">,</span> <span class="s">'tokens'</span><span class="p">:</span> <span class="n">utils</span><span class="o">.</span><span class="n">simple_preprocess</span><span class="p">(</span><span class="n">text</span><span class="p">)}</span>
<span class="gp">>>> </span> <span class="k">for</span> <span class="n">num</span><span class="p">,</span> <span class="n">text</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">texts</span><span class="p">)]</span>
</pre></div>
</div>
<p>Since corpora are allowed to be arbitrarily large, it is
recommended client splits them into smaller chunks before uploading them to the server:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">utils</span><span class="o">.</span><span class="n">upload_chunked</span><span class="p">(</span><span class="n">server</span><span class="p">,</span> <span class="n">corpus</span><span class="p">,</span> <span class="n">chunksize</span><span class="o">=</span><span class="mi">1000</span><span class="p">)</span> <span class="c"># send 1k docs at a time</span>
</pre></div>
</div>
</div>
<div class="section" id="wait-upload-what-where">
<h2>Wait, upload what, where?<a class="headerlink" href="#wait-upload-what-where" title="Permalink to this headline">¶</a></h2>
<p>If you use the similarity service object (instance of <tt class="xref py py-class docutils literal"><span class="pre">simserver.SessionServer</span></tt>) in
your code directly—no remote access—that’s perfectly fine. Using the service remotely, from a different process/machine, is an
option, not a necessity.</p>
<p>Document similarity can also act as a long-running service, a daemon process on a separate machine. In that
case, I’ll call the service object a <em>server</em>.</p>
<p>But let’s start with a local object. Open your <a class="reference external" href="http://ipython.org/">favourite shell</a> and:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">from</span> <span class="nn">gensim</span> <span class="kn">import</span> <span class="n">utils</span>
<span class="gp">>>> </span><span class="kn">from</span> <span class="nn">simserver</span> <span class="kn">import</span> <span class="n">SessionServer</span>
<span class="gp">>>> </span><span class="n">service</span> <span class="o">=</span> <span class="n">SessionServer</span><span class="p">(</span><span class="s">'/tmp/my_server/'</span><span class="p">)</span> <span class="c"># or wherever</span>
</pre></div>
</div>
<p>That initialized a new service, located in <cite>/tmp/my_server</cite> (you need write access rights to that directory).</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The service is fully defined by the content of its location directory (“<cite>/tmp/my_server/</cite>”).
If you use an existing location, the service object will resume
from the index found there. Also, to “clone” a service, just copy that
directory somewhere else. The copy will be a fully working duplicate of the
original service.</p>
</div>
</div>
<div class="section" id="model-training">
<h2>Model training<a class="headerlink" href="#model-training" title="Permalink to this headline">¶</a></h2>
<p>We can start indexing right away:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
<span class="go">AttributeError: must initialize model for /tmp/my_server/b before indexing documents</span>
</pre></div>
</div>
<p>Oops, we can not. The service indexes documents in a semantic representation, which
is different to the plain text we give it. We must teach the service how to convert
between plain text and semantics first:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">corpus</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s">'lsi'</span><span class="p">)</span>
</pre></div>
</div>
<p>That was easy. The <cite>method=’lsi’</cite> parameter meant that we trained a model for
<a class="reference external" href="http://en.wikipedia.org/wiki/Latent_semantic_indexing">Latent Semantic Indexing</a>
and default dimensionality (400) over a <a class="reference external" href="http://en.wikipedia.org/wiki/Tf–idf">tf-idf</a>
representation of our little <cite>corpus</cite>, all automatically. More on that later.</p>
<p>Note that for the semantic model to make sense, it should be trained
on a corpus that is:</p>
<ul class="simple">
<li>Reasonably similar to the documents you want to index later. Training on a corpus
of recipes in French when all indexed documents will be about programming in English
will not help.</li>
<li>Reasonably large (at least thousands of documents), so that the statistical analysis has
a chance to kick in. Don’t use my example corpus here of 9 documents in production O_o</li>
</ul>
</div>
<div class="section" id="indexing-documents">
<h2>Indexing documents<a class="headerlink" href="#indexing-documents" title="Permalink to this headline">¶</a></h2>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span> <span class="c"># index the same documents that we trained on...</span>
</pre></div>
</div>
<p>Indexing can happen over any documents, but I’m too lazy to create another example corpus, so we index the same 9 docs used for training.</p>
<p>Delete documents with:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">delete</span><span class="p">([</span><span class="s">'doc_5'</span><span class="p">,</span> <span class="s">'doc_8'</span><span class="p">])</span> <span class="c"># supply a list of document ids to be removed from the index</span>
</pre></div>
</div>
<p>When you pass documents that have the same id as some already indexed document,
the indexed document is overwritten by the new input (=only the latest counts;
document ids are always unique per service):</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">[:</span><span class="mi">3</span><span class="p">])</span> <span class="c"># overall index size unchanged (just 3 docs overwritten)</span>
</pre></div>
</div>
<p>The index/delete/overwrite calls can be arbitrarily interspersed with queries.
You don’t have to index <strong>all</strong> documents first to start querying, indexing can be incremental.</p>
</div>
<div class="section" id="querying">
<h2>Querying<a class="headerlink" href="#querying" title="Permalink to this headline">¶</a></h2>
<p>There are two types of queries:</p>
<ol class="arabic">
<li><p class="first">by id:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">print</span> <span class="n">service</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="s">'doc_0'</span><span class="p">)</span>
<span class="go">[('doc_0', 1.0, None), ('doc_2', 0.30426699, None), ('doc_1', 0.25648531, None), ('doc_3', 0.25480536, None)]</span>
</pre></div>
</div>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">print</span> <span class="n">service</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="s">'doc_5'</span><span class="p">)</span> <span class="c"># we deleted doc_5 and doc_8, remember?</span>
<span class="go">ValueError: document 'doc_5' not in index</span>
</pre></div>
</div>
<p>In the resulting 3-tuples, <cite>doc_n</cite> is the document id we supplied during indexing,
<cite>0.30426699</cite> is the similarity of <cite>doc_n</cite> to the query, but what’s up with that <cite>None</cite>, you ask?
Well, you can associate each document with a “payload”, during indexing.
This payload object (anything pickle-able) is later returned during querying.
If you don’t specify <cite>doc[‘payload’]</cite> during indexing, queries simply return <cite>None</cite> in the result tuple, as in our example here.</p>
</li>
<li><p class="first">or by document (using <cite>document[‘tokens’]</cite>; id is ignored in this case):</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">doc</span> <span class="o">=</span> <span class="p">{</span><span class="s">'tokens'</span><span class="p">:</span> <span class="n">utils</span><span class="o">.</span><span class="n">simple_preprocess</span><span class="p">(</span><span class="s">'Graph and minors and humans and trees.'</span><span class="p">)}</span>
<span class="gp">>>> </span><span class="k">print</span> <span class="n">service</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="n">doc</span><span class="p">,</span> <span class="n">min_score</span><span class="o">=</span><span class="mf">0.4</span><span class="p">,</span> <span class="n">max_results</span><span class="o">=</span><span class="mi">50</span><span class="p">)</span>
<span class="go">[('doc_7', 0.93350589, None), ('doc_3', 0.42718196, None)]</span>
</pre></div>
</div>
</li>
</ol>
</div>
<div class="section" id="remote-access">
<h2>Remote access<a class="headerlink" href="#remote-access" title="Permalink to this headline">¶</a></h2>
<p>So far, we did everything in our Python shell, locally. I very much like <a class="reference external" href="http://irmen.home.xs4all.nl/pyro/">Pyro</a>,
a pure Python package for Remote Procedure Calls (RPC), so I’ll illustrate remote
service access via Pyro. Pyro takes care of all the socket listening/request routing/data marshalling/thread
spawning, so it saves us a lot of trouble.</p>
<p>To create a similarity server, we just create a <tt class="xref py py-class docutils literal"><span class="pre">simserver.SessionServer</span></tt> object and register it
with a Pyro daemon for remote access. There is a small <a class="reference external" href="https://github.com/piskvorky/gensim-simserver/blob/master/simserver/run_simserver.py">example script</a>
included with simserver, run it with:</p>
<div class="highlight-python"><pre>$ python -m simserver.run_simserver /tmp/testserver</pre>
</div>
<p>You can just <cite>ctrl+c</cite> to terminate the server, but leave it running for now.</p>
<p>Now open your Python shell again, in another terminal window or possibly on another machine, and:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">Pyro4</span>
<span class="gp">>>> </span><span class="n">service</span> <span class="o">=</span> <span class="n">Pyro4</span><span class="o">.</span><span class="n">Proxy</span><span class="p">(</span><span class="n">Pyro4</span><span class="o">.</span><span class="n">locateNS</span><span class="p">()</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="s">'gensim.testserver'</span><span class="p">))</span>
</pre></div>
</div>
<p>Now <cite>service</cite> is only a proxy object: every call is physically executed wherever
you ran the <cite>run_server.py</cite> script, which can be a totally different computer
(within a network broadcast domain), but you don’t even know:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="k">print</span> <span class="n">service</span><span class="o">.</span><span class="n">status</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">other_corpus</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">find_similar</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
<span class="gp">>>> </span><span class="o">...</span>
</pre></div>
</div>
<p>It is worth mentioning that Irmen, the author of Pyro, also released
<a class="reference external" href="http://irmen.home.xs4all.nl/pyrolite/">Pyrolite</a> recently. That is a package
which allows you to create Pyro proxies also from Java and .NET, in addition to Python.
That way you can call remote methods from there too—the client doesn’t have to be in Python.</p>
</div>
<div class="section" id="concurrency">
<h2>Concurrency<a class=&
8BD0
quot;headerlink" href="#concurrency" title="Permalink to this headline">¶</a></h2>
<p>Ok, now it’s getting interesting. Since we can access the service remotely, what
happens if multiple clients create proxies to it at the same time? What if they
want to modify the server index at the same time?</p>
<p>Answer: the <cite>SessionServer</cite> object is thread-safe, so that when each client spawns a request
thread via Pyro, they don’t step on each other’s toes.</p>
<p>This means that:</p>
<ol class="arabic simple">
<li>There can be multiple simultaneous <cite>service.find_similar</cite> queries (or, in
general, multiple simultaneus calls that are “read-only”).</li>
<li>When two clients issue modification calls (<cite>index</cite>/<cite>train</cite>/<cite>delete</cite>/<cite>drop_index</cite>/...)
at the same time, an internal lock serializes them – the later call has to wait.</li>
<li>While one client is modifying the index, all other clients’ queries still see
the original index. Only once the modifications are committed do they become
“visible”.</li>
</ol>
</div>
<div class="section" id="what-do-you-mean-visible">
<h2>What do you mean, visible?<a class="headerlink" href="#what-do-you-mean-visible" title="Permalink to this headline">¶</a></h2>
<p>The service uses transactions internally. This means that each modification is
done over a clone of the service. If the modification session fails for whatever
reason (exception in code; power failure that turns off the server; client unhappy
with how the session went), it can be rolled back. It also means other clients can
continue querying the original index during index updates.</p>
<p>The mechanism is hidden from users by default through auto-committing (it was already happening
in the examples above too), but auto-committing can be turned off explicitly:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">set_autosession</span><span class="p">(</span><span class="bp">False</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
<span class="go">RuntimeError: must open a session before modifying SessionServer</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">open_session</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">doc_ids</span><span class="p">)</span>
<span class="gp">>>> </span><span class="o">...</span>
</pre></div>
</div>
<p>None of these changes are visible to other clients, yet. Also, other clients’
calls to index/train/etc will block until this session is committed/rolled back—there
cannot be two open sessions at the same time.</p>
<p>To end a session:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">rollback</span><span class="p">()</span> <span class="c"># discard all changes since open_session()</span>
</pre></div>
</div>
<p>or:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">>>> </span><span class="n">service</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span> <span class="c"># make changes public; now other clients can see changes/acquire the modification lock</span>
</pre></div>
</div>
</div>
<div class="section" id="other-stuff">
<h2>Other stuff<a class="headerlink" href="#other-stuff" title="Permalink to this headline">¶</a></h2>
<p>TODO Custom document parsing (in lieu of <cite>utils.simple_preprocess</cite>). Different models (not just <cite>lsi</cite>). Optimizing the index with <cite>service.optimize()</cite>.
TODO add some hard numbers; example tutorial for some bigger collection, e.g. for <a class="reference external" href="http://aura.fi.muni.cz:8080/">arxiv.org</a> or wikipedia.</p>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li><a href="index.html">Gensim home</a>| </li>
<li><a href="tutorial.html">Tutorials</a>| </li>
<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>
<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>
<li><a href="apiref.html">API reference</a>»</li>
</ul>
</div>
<div class="footer">
© Copyright 2009-2012, Radim Řehůřek <radimrehurek(at)seznam.cz>.
Last updated on Sep 16, 2012.
</div>
</body>
</html>