gensim/docs/intro.html at 8.6 · blehman/gensim

Repository navigation

History

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<head>

<title>Introduction — gensim</title>

var DOCUMENTATION_OPTIONS = {

URL_ROOT: '',

VERSION: '0.8.6',

COLLAPSE_INDEX: false,

FILE_SUFFIX: '.html',

HAS_SOURCE: true

};

</script>

<!-- twitter search widget

-->

var _gaq = _gaq || [];

_gaq.push(['_setAccount', 'UA-24066335-1']);

_gaq.push(['_trackPageview']);

(function() {

var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;

ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';

var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);

})();

</script>

</head>

<body>

<h3>Navigation</h3>

<ul>

<a href="genindex.html" title="General Index"

accesskey="I">index</a></li>

<a href="py-modindex.html" title="Python Module Index"

>modules</a> |</li>

<a href="install.html" title="Installation"

accesskey="N">next</a> |</li>

<a href="index.html" title="Gensim – Topic Modelling for Humans"

accesskey="P">previous</a> |</li>

<li><a href="index.html&qu 8000 ot;>Gensim home</a>| </li>

<li><a href="tutorial.html">Tutorials</a>| </li>

<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>

<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>

<li><a href="apiref.html">API reference</a>»</li>

</ul>

</div>

<h3><a href="index.html">Table Of Contents</a></h3>

<ul>

<li><a class="reference internal" href="#">Introduction</a><ul>

<li><a class="reference internal" href="#features">Features</a></li>

<li><a class="reference internal" href="#availability">Availability</a></li>

<li><a class="reference internal" href="#core-concepts">Core concepts</a></li>

</ul>

</li>

</ul>

<h4>Previous topic</h4>

<p class="topless"><a href="index.html"

title="previous chapter">Gensim – Topic Modelling for Humans</a></p>

<h4>Next topic</h4>

<p class="topless"><a href="install.html"

title="next chapter">Installation</a></p>

<h3>Quick search</h3>

</form>

Enter search terms or a module, class or function name.

</p>

</div>

</div>

<span id="intro"></span><h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>

<p>Gensim is a <a class="reference internal" href="#availability"><em>free</em></a> Python framework designed to automatically extract semantic

topics from documents, as efficiently (computer-wise) and painlessly (human-wise) as possible.</p>

<p>Gensim aims at processing raw, unstructured digital texts (“<em>plain text</em>”).

The algorithms in <cite>gensim</cite>, such as <strong>Latent Semantic Analysis</strong>, <strong>Latent Dirichlet Allocation</strong> or <strong>Random Projections</strong>,

discover semantic structure of documents, by examining word statistical co-occurrence patterns within a corpus of training documents.

These algorithms are unsupervised, which means no human input is necessary – you only need a corpus of plain text documents.</p>

<p>Once these statistical patterns are found, any plain text documents can be succinctly

expressed in the new, semantic representation, and queried for topical similarity

against other documents.</p>

<p class="last">If the previous paragraphs left you confused, you can read more about the <a class="reference external" href="http://en.wikipedia.org/wiki/Vector_space_model">Vector

Space Model</a> and <a class="reference external" href="http://en.wikipedia.org/wiki/Latent_semantic_indexing">unsupervised

document analysis</a> on Wikipedia.</p>

</div>

<span id="design"></span><h2>Features<a class="headerlink" href="#features" title="Permalink to this headline">¶</a></h2>

<li><strong>Memory independence</strong> – there is no need for the whole training corpus to

reside fully in RAM at any one time (can process large, web-scale corpora).</li>

<li>Efficient implementations for several popular vector space algorithms,

including <strong>Tf-Idf</strong>, distributed incremental <strong>Latent Semantic Analysis</strong>,

distributed incremental <strong>Latent Dirichlet Allocation (LDA)</strong> or <strong>Random Projection</strong>; adding new ones is easy (really!).</li>

<li>I/O wrappers and converters around <strong>several popular data formats</strong>.</li>

<li><strong>Similarity queries</strong> for documents in their semantic representation.</li>

</ul>

<p>Creation of <cite>gensim</cite> was motivated by a perceived lack of available, scalable software

frameworks that realize topic modelling, and/or their overwhelming internal complexity (hail java!).

You can read more about the motivation in our <a class="reference external" href="lrec2010_final.pdf">LREC 2010 workshop paper</a>.

If you want to cite <cite>gensim</cite> in your own work, please refer to that article (<a class="reference external" href="bibtex_gensim.bib">BibTeX</a>).</p>

<p>You’re welcome to share your results on the <a class="reference external" href="https://groups.google.com/group/gensim">mailing list</a>,

so others can learn from your success :)</p>

<p>The <strong>principal design objectives</strong> behind <cite>gensim</cite> are:</p>

<li>Straightforward interfaces and low API learning curve for developers. Good for prototyping.</li>

<li>Memory independence with respect to the size of the input corpus; all intermediate

steps and algorithms operate in a streaming fashion, accessing one document

at a time.</li>

</ol>

<p class="last">If you’re interested in document indexing/similarity retrieval, I also maintain a higher-level package

of <a class="reference external" href="http://pypi.python.org/pypi/simserver">document similarity server</a>. It uses gensim internally.</p>

</div>

<span id="id1"></span><h2>Availability<a class="headerlink" href="#availability" title="Permalink to this headline">¶</a></h2>

<p>Gensim is licensed under the OSI-approved <a class="reference external" href="http://www.gnu.org/licenses/lgpl.html">GNU LPGL license</a>

and can be downloaded either from its <a class="reference external" href="https://github.com/piskvorky/gensim/">github repository</a>

or from the <a class="reference external" href="http://pypi.python.org/pypi/gensim">Python Package Index</a>.</p>

<p class="last">See the <a class="reference internal" href="install.html"><em>install</em></a> page for more info on <cite>gensim</cite> deployment.</p>

</div>

</div> A56B ;

<h2>Core concepts<a class="headerlink" href="#core-concepts" title="Permalink to this headline">¶</a></h2>

<p>The whole gensim package revolves around the concepts of <a class="reference internal" href="#term-corpus"><em class="xref std std-term">corpus</em></a>, <a class="reference internal" href="#term-vector"><em class="xref std std-term">vector</em></a> and

<a class="reference internal" href="#term-model"><em class="xref std std-term">model</em></a>.</p>

<dt id="term-corpus">Corpus</dt>

<dd>A collection of digital documents. This collection is used to automatically

infer structure of the documents, their topics etc. For

this reason, the collection is also called a <em>training corpus</em>. The inferred

latent structure can be later used to assign topics to new documents, which did

not appear in the training corpus.

No human intervention (such as tagging the documents by hand, or creating

other metadata) is required.</dd>

<dt id="term-vector">Vector</dt>

<dd><p class="first">In the Vector Space Model (VSM), each document is represented by an

array of features. For example, a single feature may be thought of as a

question-answer pair:</p>

<li>How many times does the word <em>splonge</em> appear in the document? Zero.</li>

<li>How many paragraphs does the document consist of? Two.</li>

<li>How many fonts does the document use? Five.</li>

</ol>

<p>The question is usually represented only by its integer id (such as <cite>1</cite>, <cite>2</cite> and <cite>3</cite> here),

so that the

representation of this document becomes a series of pairs like <tt class="docutils literal"><span class="pre">(1,</span> <span class="pre">0.0),</span> <span class="pre">(2,</span> <span class="pre">2.0),</span> <span class="pre">(3,</span> <span class="pre">5.0)</span></tt>.

If we know all the questions in advance, we may leave them implicit

and simply write <tt class="docutils literal"><span class="pre">(0.0,</span> <span class="pre">2.0,</span> <span class="pre">5.0)</span></tt>.

This sequence of answers can be thought of as a high-dimensional (in this case 3-dimensional)

<em>vector</em>. For practical purposes, only questions to which the answer is (or

can be converted to) a single real number are allowed.</p>

<p class="last">The questions are the same for each document, so that looking at two

vectors (representing two documents), we will hopefully be able to make

conclusions such as “The numbers in these two vectors are very similar, and

therefore the original documents must be similar, too”. Of course, whether

such conclusions correspond to reality depends on how well we picked our questions.</p>

</dd>

<dt id="term-sparse-vector">Sparse vector</dt>

<dd><p class="first">Typically, the answer to most questions will be <tt class="docutils literal"><span class="pre">0.0</span></tt>. To save space,

we omit them from the document’s representation, and write only <tt class="docutils literal"><span class="pre">(2,</span> <span class="pre">2.0),</span>

<span class="pre">(3,</span> <span class="pre">5.0)</span></tt> (note the missing <tt class="docutils literal"><span class="pre">(1,</span> <span class="pre">0.0)</span></tt>).

Since the set of all questions is known in advance, all the missing features

in a sparse representation of a document can be unambiguously resolved to zero, <tt class="docutils literal"><span class="pre">0.0</span></tt>.</p>

<p class="last">Gensim is specific in that it doesn’t prescribe any specific corpus format;

a corpus is anything that, when iterated over, successively yields these sparse vectors.

For example, <cite>set([(2, 2.0), (3, 5.0)], ([0, -1.0], [3, -1.0]))</cite> is a trivial

corpus of two documents, each with two non-zero <cite>feature-answer</cite> pairs.</p>

</dd>

<dt id="term-model">Model</dt>

<dd>For our purposes, a model is a transformation from one document representation

to another (or, in other words, from one vector space to another).

Both the initial and target representations are

still vectors – they only differ in what the questions and answers are.

The transformation is automatically learned from the traning <a class="reference internal" href="#term-corpus"><em class="xref std std-term">corpus</em></a>, without human

supervision, and in hopes that the final document representation will be more compact

and more useful: with similar documents having similar representations.</dd>

</dl>

<p class="last">For some examples on how this works out in code, go to <a class="reference internal" href="tutorial.html"><em>tutorials</em></a>.</p>

</div>

</div>

<h3>Navigation</h3>

<ul>

<a href="genindex.html" title="General Index"

>index</a></li>

<a href="py-modindex.html" title="Python Module Index"

>modules</a> |</li>

<a href="install.html" title="Installation"

>next</a> |</li>

<a href="index.html" title="Gensim – Topic Modelling for Humans"

>previous</a> |</li>

<li><a href="index.html">Gensim home</a>| </li>

<li><a href="tutorial.html">Tutorials</a>| </li>

<li><a href="http://groups.google.com/group/gensim">Support</a>| </li>

<li><a href="https://github.com/piskvorky/gensim/wiki">Contribute</a>| </li>

<li><a href="apiref.html">API reference</a>»</li>

</ul>

</div>

Last updated on Sep 16, 2012.

</div>

</body>

</html>

FilesExpand file tree

intro.html

Latest commit

History

intro.html

File metadata and controls