UNPKG

lexemic

Version:

Tools for working with human language data.

bertrandk/lexemic

117 lines (92 loc) • 4.29 kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
# lexemic [![NPM version](https://badge.fury.io/js/lexemic.png)](http://badge.fury.io/js/lexemic) ## Requirements * Node.js (v0.10.17+) * npm (v1.3.9+) ## Install npm install lexemic -g ## Description Tools for working with human language data. ## Features * Sentiment analysis * Stemming * Tokenization * Statistical analysis ## Usage ``` $ lexemic [command] [implementation] [target...] ``` NOTE: The `target` may be an inline string or the path to a text file encoded as UTF-8. ### Sentiment analysis ``` $ lexemic sentiment "I am mad at you." # => { :score -1, :comparative -0.25, :positive { :score 0, :comparative 0, :words () }, :negative { :score 1, :comparative 0.25, :words (mad) } } ``` Sentiment analysis attempts to determine the affective state of the speaker or the writer. The default implementation returns an [EDN](https://github.com/edn-format/edn) map of this analysis. The `:score` represents the number of emotive words in the text while the `:comparative` rates the occurrence of these words with regards to the length of the text. The nested values (i.e. those under `:positive` and `:negative` ) provide a list of matched `:words` and take only into account their respective affectivity. The top level values incorporate both affective states – returning negative values for texts with overall negative affects and positive values for texts with overall positive affects. ### Stemming ``` $ lexemic stem "My education has been educational" # => #{"My" "educ" "ha" "been"} ``` Stemming attempts to reduce inflected words to their stem. This is useful in reducing your working set of words and expanding possible search matches. The default implementation uses the porter algorithm, though you may explicitly specify an implementation; `-p` and `-porter` for the porter algorithm (standard and gentle) or `-l` and `-lancaster` for the lancaster algorithm (much more aggressive). This command returns an [EDN](https://github.com/edn-format/edn) set of the reduced working set. ### Tokenization ``` $ lexemic tokenize "This is a sample sentence." # => ["This" "is" "a" "sample" "sentence" "."] ``` Tokenization attempts to break a text into its desired constituent parts – typically, that is into words. This command returns an [EDN](https://github.com/edn-format/edn) vector of the result of this process. ### Statistical analysis #### Levenshtein distance ``` $ lexemic distance "This is a sentence." "This is a similar sentence." # => 8 ``` Levenshtein distance measures the distance between two strings of text, often two documents. Informally, this distance corresponds to the minimum number of single-character edits required to change one string into the other (e.g. in the above example, '8' represents the insertion of 1 space and the 7 letters of the word 'similar'). This command returns an integer representing the result of this process. ## Issues If you need help, find a bug, want to request a feature or want to contribute, please [create an issue](https://github.com/bertrandk/lexemic/issues/new). ## Copyright Copyright (c) 2013 Bertrand Karerangabo See `LICENSE.txt` for details.