GMU:The Hidden Layer:Topics - Revision history

Fbonowski at 13:10, 9 May 2017

2017-05-09T13:10:49Z

← Older revision		Revision as of 13:10, 9 May 2017
Line 5:		Line 5:
	The whole process goes through a number of stages:		The whole process goes through a number of stages:

	=== 1. ~~The text~~ corpus ===		=== 1. Text corpus ===
	This is the raw data used for learning. Determines language, the topics that are covered and the semantics.		This is the raw data used for learning. Determines language, the topics that are covered and the semantics.
	Typical sources are Wikipedia and news articles.		Typical sources are Wikipedia and news articles.
	=== 2. ~~The tokens~~ ===
			=== 2. Tokens ===
	The corpus is split into words. These might be processed further, i.e. to clean up junk, or taking flections (i.e. verb forms) into account.		The corpus is split into words. These might be processed further, i.e. to clean up junk, or taking flections (i.e. verb forms) into account.

	=== 3. Contexts ===		=== 3. Contexts ===
	The words are grouped into contexts. These might be all words in a sentence, a certain number of words in a neighborhood or words that somehow relate to each other gramatically (as in "dependency based word embeddings" [https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/]).		The words are grouped into contexts. These might be all words in a sentence, a certain number of words in a neighborhood (aka. "bag of words") or words that somehow relate to each other gramatically (as in "dependency based word embeddings" [https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/]).

	=== 4. The algorithm ===		=== 4. The algorithm ===
	Different algorithms can be used to map the relationship between words and their context into a vector space. The main contesters are...		Different algorithms can be used to map the relationship between words and their context into a vector space. The main contesters are...
	* '''Word2vec''' by Google, uses Neural Networks		* '''Word2vec''' by Google, uses Neural Networks
	* '''~~FastwordMade~~''' by Facebook based on word2vec. Splits words into smaller particles in order to capture capturing syntactic relations (like apparent ---> apparently). Explained here: [https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]		* '''Fastword''' by Facebook based on word2vec. Splits words into smaller particles in order to capture capturing syntactic relations (like apparent --> apparently). Explained here: [https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]. Needs a lot of memory.
	* '''GloVe''' by the Natural language processing group in standford [https://nlp.stanford.edu/projects/glove/]. Uses more conventional math instead of Neural Network "Black Magic" [https://www.quora.com/How-is-GloVe-different-from-word2vec]. ~~Seems~~ to perform ~~just slightly~~ less ~~well than Word2vec~~ and ~~FastWord~~.		* '''GloVe''' by the Natural language processing group in standford [https://nlp.stanford.edu/projects/glove/]. Uses more conventional math instead of Neural Network "Black Magic" [https://www.quora.com/How-is-GloVe-different-from-word2vec].

			The different algorithms seem to perform quite similar, and results depend on the benchmark and training data. Word2Vec seems to be a little less memory hungry, though.
			=== 5. Keyed Vecors ===

			Here comes the '''Good news''': All of the algorithms provide a table with words and and their positions in vector space... So '''all you need is that table'''!

			Fastvec is special in beeing able to match also on words that it hasn't seen before... but we probably don't even need that...

			==== pre trained models ====
			Here is a collection of Words->Vector tables ("models") that other people have created from big corpuses. This is what you probably want:

			* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter, '''a good mix''']
			* [https://github.com/Kyubyong/wordvectors https://github.com/Kyubyong/wordvectors: Word2Vec and FastText, '''Multiple languages''', no english, trained on Wikipedia]
			* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia, HUGE files]
			* [https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ an interesting approach that gives similarities between syntaktically equivalent words]

			In order to convert from GloVe to Word2Vec tables, the following script can be used:
			[[https://radimrehurek.com/gensim/scripts/glove2word2vec.html]]

	== Installation + getting started: ==		== Installation + getting started: ==
Line 51:		Line 71:
	==GloVe==		==GloVe==
	Invented by the Natural language processing group in standford [https://nlp.stanford.edu/projects/glove/]. Uses more conventional math instead of Neural Network "Black Magic" [https://www.quora.com/How-is-GloVe-different-from-word2vec]. Seems to perform just slightly less well than Word2vec and FastWord.		Invented by the Natural language processing group in standford [https://nlp.stanford.edu/projects/glove/]. Uses more conventional math instead of Neural Network "Black Magic" [https://www.quora.com/How-is-GloVe-different-from-word2vec]. Seems to perform just slightly less well than Word2vec and FastWord.

	~~== pre trained models ==~~

	* [https://github.com/Kyubyong/wordvectors https://github.com/Kyubyong/wordvectors: Word2Vec and FastText, Multiple languages, no english, trained on Wikipedia]
	* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter]
	* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia]
	* [https://radimrehurek.com/gensim/scripts/glove2word2vec.html https://radimrehurek.com/gensim/scripts/glove2word2vec.html convert between GloVe and Word2Vec Format]
	* [https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ an interesting approach that gives similarities between syntaktically equivalent words]

Fbonowski at 18:59, 8 May 2017

2017-05-08T18:59:41Z

@@ Line 1: / Line 1: @@
 == General Information on word embeddings ==
-For a general explanation look here:
+Word embeddings associate words with vectors in a high-dimensional space. Words that are close together in that space are more likely to occur in close proximity in a test than words which are far apart. See this article for details: [https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/]
 [https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/]
 ==Word2vec==
 Included in the ''gensim'' package.

Fbonowski at 17:53, 8 May 2017

2017-05-08T17:53:16Z

← Older revision		Revision as of 17:53, 8 May 2017
Line 3:		Line 3:
	For a general explanation look here:		For a general explanation look here:
	[https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/]		[https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/]

			As wordvector algorithms
	==Word2vec==		==Word2vec==
	Made by Google, uses Neural Net, performs good on semantics.		Made by Google, uses Neural Net, performs good on semantics.
Line 42:		Line 44:
	* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter]		* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter]
	* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia]		* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia]
			* [https://radimrehurek.com/gensim/scripts/glove2word2vec.html https://radimrehurek.com/gensim/scripts/glove2word2vec.html convert between GloVe and Word2Vec Format]
			* [https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ an interesting approach that gives similarities between syntaktically equivalent words]

Fbonowski at 11:35, 8 May 2017

2017-05-08T11:35:59Z

← Older revision		Revision as of 11:35, 8 May 2017
Line 35:		Line 35:

	==GloVe==		==GloVe==
	Invented by the Natural language processing group in standford. [https://nlp.stanford.edu/projects/glove/]Uses more conventional math instead of Neural Network "Black Magic". Seems to perform ~~very~~ slightly less well than Word2vec and FastWord.		Invented by the Natural language processing group in standford [https://nlp.stanford.edu/projects/glove/]. Uses more conventional math instead of Neural Network "Black Magic" [https://www.quora.com/How-is-GloVe-different-from-word2vec]. Seems to perform just slightly less well than Word2vec and FastWord.

	== pre trained models ==		== pre trained models ==

Fbonowski at 11:28, 8 May 2017

2017-05-08T11:28:51Z

← Older revision		Revision as of 11:28, 8 May 2017
Line 7:		Line 7:

	=== Installation + getting started: ===		=== Installation + getting started: ===
			Included in the ''gensim'' package.

			To install, just type

	<code>pip install gensim</code><br>		<code>pip install gensim</code><br>

			into a command window.

	Here are some of the things you can do with the model: [http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python]<br>		Here are some of the things you can do with the model: [http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python]<br>
	Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].		Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].
Line 17:		Line 24:

	=== Installation + getting started: ===		=== Installation + getting started: ===
	Included in ~~Gensim~~. ~~Couldn't test yet due to memory constraints~~.		Included in the ''gensim'' package.

			To install, just type

			<code>pip install gensim</code><br>

			into a command window.

	Documentation is here: [https://radimrehurek.com/gensim/models/wrappers/fasttext.html]		Documentation is here: [https://radimrehurek.com/gensim/models/wrappers/fasttext.html]

Fbonowski at 11:27, 8 May 2017

2017-05-08T11:27:18Z

← Older revision		Revision as of 11:27, 8 May 2017
Line 5:		Line 5:
	==Word2vec==		==Word2vec==
	Made by Google, uses Neural Net, performs good on semantics.		Made by Google, uses Neural Net, performs good on semantics.

	=== Installation + getting started: ===		=== Installation + getting started: ===
	<code>pip install gensim</code><br>		<code>pip install gensim</code><br>
Line 10:		Line 11:
	Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].		Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].
	==Fastword==		==Fastword==
	Made by ~~Facebbok~~ based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:		Made by Facebook based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:
	[https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]<br>		[https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]<br>

	Pretrained model files are HUGE		Pretrained model files are HUGE - this will be a problem on computers with less than 16GB Memory

			=== Installation + getting started: ===
			Included in Gensim. Couldn't test yet due to memory constraints.
			Documentation is here: [https://radimrehurek.com/gensim/models/wrappers/fasttext.html]

	==GloVe==		==GloVe==
			Invented by the Natural language processing group in standford. [https://nlp.stanford.edu/projects/glove/]Uses more conventional math instead of Neural Network "Black Magic". Seems to perform very slightly less well than Word2vec and FastWord.

	== pre trained models ==		== pre trained models ==

	* [https://github.com/Kyubyong/wordvectors https://github.com/Kyubyong/wordvectors: Word2Vec and FastText, Multiple languages, no english, trained on Wikipedia]		* [https://github.com/Kyubyong/wordvectors https://github.com/Kyubyong/wordvectors: Word2Vec and FastText, Multiple languages, no english, trained on Wikipedia]
	* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter]		* [https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models https://github.com/3Top/word2vec-api Mostly GloVe, some word2vec, English, Trained on News, Wikipedia, Twitter]
	* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia]		* [https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md: Fasttext, all imaginable languages, trained on Wikipedia]

Fbonowski: /* Fastword */

2017-05-08T10:10:24Z

Fastword

← Older revision		Revision as of 10:10, 8 May 2017
Line 11:		Line 11:
	==Fastword==		==Fastword==
	Made by Facebbok based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:		Made by Facebbok based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:
	[https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]		[https://rare-technologies.com/fasttext-and-gensim-word-embeddings/]<br>

	Pretrained model files are HUGE		Pretrained model files are HUGE

Fbonowski at 10:09, 8 May 2017

2017-05-08T10:09:35Z

← Older revision		Revision as of 10:09, 8 May 2017
Line 6:		Line 6:
	Made by Google, uses Neural Net, performs good on semantics.		Made by Google, uses Neural Net, performs good on semantics.
	=== Installation + getting started: ===		=== Installation + getting started: ===
	<code>pip install gensim</code>\\		<code>pip install gensim</code><br>
	Here are some of the things you can do with the model: [http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python]\\		Here are some of the things you can do with the model: [http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python]<br>
	Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].		Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].
	==Fastword==		==Fastword==

Fbonowski at 10:08, 8 May 2017

2017-05-08T10:08:52Z

← Older revision		Revision as of 10:08, 8 May 2017
Line 6:		Line 6:
	Made by Google, uses Neural Net, performs good on semantics.		Made by Google, uses Neural Net, performs good on semantics.
	=== Installation + getting started: ===		=== Installation + getting started: ===
	<code>pip install gensim</code>		<code>pip install gensim</code>\\
			Here are some of the things you can do with the model: [http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python]\\
			Here is a bit of background information an an explanation how to train your own models: [https://rare-technologies.com/word2vec-tutorial/].
	==Fastword==		==Fastword==
	Made by Facebbok based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:		Made by Facebbok based on word2vec. Better at capturing syntactic relations (like apparent ---> apparently) see here:

Fbonowski at 09:55, 8 May 2017

2017-05-08T09:55:04Z

← Older revision		Revision as of 09:55, 8 May 2017
Line 6:		Line 6:
	Made by Google, uses Neural Net, performs good on semantics.		Made by Google, uses Neural Net, performs good on semantics.
	=== Installation + getting started: ===		=== Installation + getting started: ===
	pip install gensim		<code>pip install gensim</code>

	==Fastword==		==Fastword==