vui-ad-hoc-alexa-recognizer
Version:
Provides natural language understanding/processing to enable easy implementation of chat bots and voice services. High performance run time in only 2 lines of code - 'require' to include it, and the call to process the text. These can run anywhere Node.js
1,180 lines (991 loc) • 109 kB
Markdown
[](https://travis-ci.org/RationalAnimal/vui-ad-hoc-alexa-recognizer)
[](https://npm-stat.com/charts.html?package=vui-ad-hoc-alexa-recognizer)
[](http://open-oss.com)
[](https://www.npmjs.com/package/vui-ad-hoc-alexa-recognizer)
[](http://isitmaintained.com/project/RationalAnimal/vui-ad-hoc-alexa-recognizer "Average time to resolve an issue")
[](http://isitmaintained.com/project/RationalAnimal/vui-ad-hoc-alexa-recognizer "Percentage of issues still open")
[](https://www.patreon.com/RationalAnimal)
# vui-ad-hoc-alexa-recognizer
Provides natural language understanding/processing capability to enable easy implementation of chat bots and voice services.
High performance run time in only 2 lines of code - 'require' to include it, and the call to process the text.
These can run anywhere Node.js is running - backend, browser, mobile apps, etc with or without internet connection.
Has a rich set of built in intents and extensible slots (equivalent to Alexa's),
custom slots (both list based and regular expression based), synonyms, slot flags, parametrized flags, transformation functions,
SoundEx matching, wild card matching, option lists, text equivalents sets, mix-in post processing, sentiment analysis,
unlimited sets of recognizers to build large
segmented apps, domains with state specific processing, builtin and custom chainable responders, sub-domains (trusted and
non-trusted), etc.
# What's in the name
You may be wondering why such an odd name. Glad you asked, here is the explanation:
1. vui - stands for "voice user interface" - because this module allows building skills/apps that have voice user interface
2. ad-hoc - because this module creates a pre-configured run time for specific set(s) of intents (requiring no further configuration at run time)
3. alexa - because this module started out by using Alexa skill configuration files (it has expanded well beyond that, but can still be used as before).
So, if you already have an Alexa skill, using this module should be very easy and fast
(and even if you don't it's still easy and fast, just a little longer to configure).
Also, you can use this module simply to ease your Alexa coding - you can configure everything using vui-ad-hoc-alexa-recognizer,
saving yourself time and effort by not having to manually enter all the variations, then use the included alexify utility to output alexa compatible files
4. recognizer - because that's what it does - "recognizes" and processes utterances to identify intents, extract slot values, and optionally provide responses and update the app state.
# Repository
This module as well as related vui modules can be found here:
https://github.com/RationalAnimal
# Tutorials, Examples, Documentation
I have finally gotten to a point where the major features are at a good spot and I can spare some time for
tutorials, examples, and documentation.
To that end I have set up a web site on GitHub pages and included the first set of tutorials, with more on the way.
Here is the website:
https://rationalanimal.github.io/vui-ad-hoc-alexa-recognizer/
## Recognizer Tutorials
The first set of tutorials deals with the lower level, i.e. recognizer functionality:
* "Hello World" and "Using Hello World" - these two tutorials together comprise the usual "Hello World" example -
one shows how to configure, generate, and test a recognizer.json file.
The other tutorial shows how to write your own code to use the generated file.
* "You Have Options... Option Lists, That Is" - this tutorial shows how to avoid having to manually enter and maintain a
large number of related utterances.
* "Great Intrinsic Value" - this one shows how to get information back from the user via BUILT IN slot types.
* "Let's Talk" - here the user is shown how to build an actual chat bot - a small app that takes input from the user,
parses it, stores some values in the state for future use, and responds to the user based on the parsed
information and state.
* "Count on it" - introduces the use of what is probably the singularly most useful built in slot type - NUMBER.
* "He Spoke Bespoke" - deals with how to define the simplest type of a custom slot - list based custom slot.
* "Known Intentions" - deals with another built in feature: shows how to use, configure, and turn off built in intents.
* "It's All the Same to Me" - demonstrates custom slots that use synonyms to simplify the code that has to deal with
multiple values that map to a smaller subset.
* "Wild (Card) Thing" - explains how to get values from the user that are not part of the "accepted" set, i.e. using
wildcard matches.
* "Express Yourself Regularly" - explains how to use regular expressions based custom slots to get user input that
matches a particular pattern rather than specific list of values.
* "That Sounds Like You" - explains how to use SOUNDEX matching to process words that sound similar. This helps with
commonly substituted words in chat bots and words that sound the same but aren't spelled the same way in voice services.
* "Six of One, Half a Dozen of the Other" - covers text equivalents sets and their use. Text equivalents feature lets you
define words and phrases that are equivalent to each other so that you need only to specify a simple utterance and
have vui-ad-hoc-alexa-recognizer match on any variation of that utterance that may result from using text equivalents.
This allows solving many different issues - from typos, to homophones, to special vocabularies, etc...
* "Would You Like Some Fries with That?" - introduces the concept of "mix in" (aka "add on") processing.
Shows how you can use mix ins to cleanly separate matching and business logic, change matching through configuration and
mix in application, add logging, and more.
## Domain Tutorials
Finally, here is the first domain tutorial:
* "Hello Domain" - this is a very simple, introductory example of a domain that doesn't use the state nor any of the
advanced features, but does respond to every handled intent without the need for any conversation specific code.
# Articles
In addition to tutorials I will from time to time publish articles related to either chatbots/voice services in general
or vui-ad-hoc-alexa-recognizer in particular (or some of both). They are located on the same website as the tutorials.
Here is the first of these:
* "Better Way of Building Better Chatbots" at https://rationalanimal.github.io/vui-ad-hoc-alexa-recognizer/Articles/BetterWayOfBuildingBetterChatbots/
# Note on licenses
The code in this project is distributed under the MIT license. Some data files
found in the builtinslottypes directory (e.g. colors.json) use values taken from Wikipedia
and thus they are licensed under Creative Commons Attribution-ShareAlike license.
Such files have appropriate attribution and license information within them.
If you don't wish to deal with these licenses, simply delete such file(s) from
the builtinslottypes directory. You will need to provide your own versions of
these files. Note that only some of those files have a
different license - you DON'T need to delete the entire directory to remove them.
Simply search in builtinslottypes directory for "license" and/or "attribution"
Also, AFINN related JSON data files in the builtinmixins directory are a modification of the original files (see
attribution within the files) distributed under Apache 2.0 license.
# Installation
```shell
npm install vui-ad-hoc-alexa-recognizer --save
```
# Summary
This module provides the ability to match user's text input (possibly from speech to text source) to an intent with
slot values as well as the ability to configure return values and state updates right within it.
There are two levels of functionality - a lower level allowing a match of a string against a single recognizer, returning
an intent match with parsed out slot values and a higher level "domain" functionality that allows configuring an
entire app, returning not just a match but results and even updating the current state.
## "Recognizer" (or lower level) functionality
npm module that provides VUI (voice user interface) special ad-hoc recognizer
designed to parse raw text from the user and map it to intents.
This could be useful in many cases. If
you already have an Alexa skill and would like to convert it to
Google assistant, or some other service, this module makes it really easy. Many skills can be converted
in less than an hour.
Also you can use this to quickly create a skill or an app even if you don't already have an Alexa skill.
You will simply need to create the
required intents, utterances, and (optionally) custom slot value files (equivalent of which
you'd have to do anyway).
It uses the same two files (intents and utterances) that are used to configure Alexa skills.
(It also supports the "beta" Alexa configuration, but I don't recommend using it yet as).
This allows easy "middleware" implementation that can be placed between
Google assistant (or other service) and the Alexa backend. If you have custom slots and
you want to use exact or SoundEx matches on those slots, then you would also need
file(s) listing these values.
Supports almost all Alexa features - built in intents, all major built in slot types,
most minor ones,
as well as "extra" features, such as the ability to do wildcard or SoundEx
matches, transforming the values before sending them to be processed, text equivalents matching (such as misspellings,
equivalent words or phrases, homophones and more) etc.
Additional configuration can be added through config file.
You can also use it without any backend service whatsoever - simply use it with your javascript
code same way you would use any other npm module. It will provide complete
utterance parsing and slot values mapping. Simply use simple branching
code (e.g. switch statement) using the intent to complete processing.
Keep in mind that many text parsing tasks can be trivially configured as intents/utterances even if you have no intention
of building a chat bot or a voice service. For example, if you wanted
to parse spelled out numbers, or even combinations of spelled out numbers and numerals, you can easily setup to do it like this:
utterances:
```text
NumberIntent {NumberSlot}
```
intents:
```json
{
"intents": [
{
"intent": "NumberIntent",
"slots": [
{
"name": "NumberSlot",
"type": "AMAZON.NUMBER"
}
]
}
]
}
```
Now, you can call the match() function, pass it any combination of spelled out and numerals, and it will return
a match on NumberIntent with the value of the NumberSlot being set to the parsed number:
```shell
node matcher.js "51 thousand 2 hundred sixty 3"
```
```json
{
"name": "NumberIntent",
"slots": {
"NumberSlot": {
"name": "NumberSlot",
"value": "51263"
}
}
}
```
Note that the Number slot is coded to be able to accept both "normal" numbers and numbers that people spell out digit by
digit or groups of digits, such as zip codes or phone numbers. So "one two three four five" will parse as "12345", etc.
This does mean that occasionally there may come up a way to parse the same expression in more than one way and the attempt
is made to make the most likely match.
Similarly, you can parse dates, etc. even if that's the only thing you want to do (e.g. you have an app where the user
can type in a date - simply use vui-ad-hoc-alexa-recognizer to parse it and return a date). Dates will match not only the
exact date specification, but strings such as "today", etc.
## "Domain" (higher level) functionality
Domains are a higher level of parsing than recognizers. Domains do use "recognizer" parsing, but add the follow abilities:
* define a list of recognizers to be used
* define application state-based conditions for using a particular match (e.g. only test against a particular recognizer if you are in a specific state)
* allow returning of results in addition to simply matching on an intent (e.g. if the user says "How are you doing?", not only will it match on a greeting intent, but also will return "Good, and you?")
* allow updating of the application state right within the matching code rather than having to write the extra code to do it (e.g. if the user says "My name is Bob" then some portion of the state will be set to "Bob" by the domain handling code)
* allow nesting of the domains. This is particularly useful as whole types of interactions can be encapsulated as domains and then reused. It also allows breaking large apps into smaller chunks, i.e. domains.
# Usage
## "Recognizer" (or lower level) functionality
It has two pieces of functionality:
* run it offline to generate a recognizer.json file that will be used in matching/parsing the text
* add two lines of code to your app/skill to use it to match the raw text at run time using the generated recognizer.json file.
Imagine you already have an Alexa skill and you would like to port it to Cortana
or Google Assistant (or even if you don't but want to create a chat bot/service from scratch).
Here are examples of files that you will have for your
Alexa skill or will need to create if you don't have any yet
(these are NOT complete files, you can find the complete sample
files in the test directory):
```shell
cat test/utterances.txt
```
```
TestIntent test
TestIntent test me
TestIntent test please
TestIntent test pretty please
TestIntent test pleeeeeeease
MinionIntent One of the minions is {MinionSlot}
MinionIntent {MinionSlot}
StateIntent {StateSlot}
StateIntent New England includes {StateSlot} as one of its states
BlahIntent here is my number {BlahSlot}, use it wisely. And here is another one {BlehSlot}, don't squander it
BlahIntent here is {BlahSlot} and {BlehSlot}
AnotherIntent First is {SomeSlot} and then there is {SomeOtherSlot}
```
```shell
cat test/intents.json
```
```json
{
"intents": [
{
"intent": "TestIntent",
"slots": []
},
{
"intent": "BlahIntent",
"slots": [
{
"name": "BlahSlot",
"type": "AMAZON.NUMBER"
},
{
"name": "BlehSlot",
"type": "AMAZON.NUMBER"
}
]
},
{
"intent": "AnotherIntent",
"slots": [
{
"name": "SomeSlot",
"type": "SOME"
},
{
"name": "SomeOtherSlot",
"type": "SOMEOTHER"
}
]
},
{
"intent": "MinionIntent",
"slots": [
{
"name": "MinionSlot",
"type": "MINIONS"
}
]
},
{
"intent": "StateIntent",
"slots": [
{
"name": "StateSlot",
"type": "AMAZON.US_STATE"
}
]
}
]
}
```
and also here is an example of a custom slot type file:
```shell
cat test/minions.txt
```
```
Bob
Steve
Stewart
```
### BETA InteractionModel.js file support
Currently, Amazon is beta testing a new GUI tools for editing skills' interaction models. This produces a single json
file that includes ALL the information for the skill - intents, utterances, custom slot types, and some "new" bits:
prompts/dialogs/confirmations. The support for this is being added to various parts of vui-ad-hoc-alexa-recognizer
(including the generator and alexifyer) and is functional. However, until Amazon finishes the beta testing, this will not be finalized and
may change.
* NOTE: AT THIS TIME I DO NOT RECOMMEND USING INTERACTION MODEL FILES - WHILE I HAVE FINISHED THE SUPPORT, I HAVE NOT TESTED IT FULLY
### Generate recognizer.json file
The first step is to generate a run time file - recognizer.json. This file has
all the information that is needed to parse user text later. To create it, run
the generator, e.g.:
```shell
node generator.js --intents test/intents.json --utterances test/utterances.txt --config test/config.json
```
(the example intents.json, utterances.txt, and config.json files are included in the test directory)
This will produce a recognizer.json in the current directory.
Additionally, there is beta support for the beta interaction model builder by Amazon. To use it specify --interactionmodel
parameter instead of --intents and --utterances:
```shell
node generator.js --interactionmodel test/interactionmodel.json --config test/config.json
```
* NOTE: AT THIS TIME I DO NOT RECOMMEND USING INTERACTION MODEL FILES - WHILE I HAVE FINISHED THE SUPPORT, I HAVE NOT TESTED IT FULLY
Note that you can use the extra features in the interactionmodel.json file just as you could with intents.json and
utterances.txt (e.g. options lists, slot flags, TRANSCEND specific slot types). Simply use alexifyutterances.js (see later) to prepare interactionmodel.json for import into Alexa
developer console.
For usage, simply run the generator without any arguments:
```shell
node generator.js
```
and the generator command will list the needed arguments, e.g.:
```shell
Usage: node /Users/ilya/AlexaProjects/vui-ad-hoc-alexa-recognizer/vui-ad-hoc-alexa-recognizer/generator.js:
--sourcebase BaseSourceDirectory that is the base for the other file references on the command line or in the config file. This will be used for both build and run time source base unless overridden by other command line arguments.
--buildtimesourcebase BuildTimeBaseSourceDirectory that is the base for the other file references on the command line or in the config file at build time. Will override --sourcebase value for build time directory, if both are supplied
--runtimesourcebase RunTimeBaseSourceDirectory that is the base for the other file references (e.g. in the config file) at run time. Will override --sourcebase value for run time directory, if both are supplied
--vuibase BaseVuiDirectory that is the location of vui-ad-hoc-alexa-recognizer. This will be used for both build and run time vui base unless overridden by other command line arguments. Defaults to ./node_modules/vui-ad-hoc-alexa-recognizer
--buildtimevuibase BuildTimeBaseVuiDirectory that is the location of vui-ad-hoc-alexa-recornizer executable files at build time. Will override --vuibase value for build time directory, if both are supplied
--runtimevuibase RunTimeBaseVuiDirectory that is the location of vui-ad-hoc-alexa-recognizer executable files at run time. Will override --vuibase value for run time directory, if both are supplied
--runtimeexebase RunTimeBaseExeDirectory that is the location of javascript executable files at run time.
--config ConfigFileName specify configuration file name, optional. If not specified default values are used.
--intents IntentsFileName specify intents file name, required. There is no point in using this without specifying this file.
--utterances UtterancesFileName specify utterances file name, optional. This is "optional" only in the sense that it CAN be omitted, but in practice it is required. There only time you would invoke this function without an utterance file argument is if your skill generates only build in intents, which would make it rather useless.
--optimizations [SINGLE-STAGE] optional. SINGLE-STAGE means no pre-matches using wildcards. Depending on the recognizer, this may be slower or faster
--suppressRecognizerDisplay does not send recognizer.json to console
```
Note here that you should already have the intents.json and utterances.txt files
as these files are used to configure the Alexa skill.
Also, you can specify how to parse built in intents in the config.json.
For example:
```json
{
"builtInIntents":[
{
"name": "AMAZON.RepeatIntent",
"enabled": false
}
]
}
```
will turn off parsing of the AMAZON.RepeatIntent.
You can also specify additional utterances for built in intents, either directly
in the config file or in an external file:
```json
{
"builtInIntents":[
{
"name": "AMAZON.StopIntent",
"enabled": true,
"extendedUtterances": ["enough already", "quit now"],
"extendedUtterancesFilename": "test/stopIntentExtendedUtterances.txt"
}
]
}
```
Similarly you can affect built in slot types using config:
```json
{
"builtInSlots": [
{
"name": "AMAZON.US_FIRST_NAME",
"extendedValues": [
"Prince Abubu"
],
"extendedValuesFilename": "test/usFirstNameExtendedValues.txt"
}
]
}
```
This will add "Prince Abubu" and whatever names are found in test/usFirstNameExtendedValues.txt
file to the list of first names recognized by the AMAZON.US_FIRST_NAME slot.
### Parse user text
The second step is to use recognizer.json file at run time to parse the user
text and produce the output json that can be used to set the intent portion of
the request json.
You only need 2 lines of code to be added to your app to use it:
```js
let recognizer = require("vui-ad-hoc-alexa-recognizer");
let parsedResult = recognizer.Recognizer.matchText("Some text to match to intent");
```
(If this is not working, check to make sure you have generated your recognizer.json
first and that it's located in the same directory where your code is.)
Note that there are additional arguments to matchText(). You can specify sorting
order, excluded intents, and a different recognizer file.
You can also use it from the command line to test your configuration and to see
the matches. For an example of how to use it (assuming you cloned the code from
GitHub and ran "npm test" to have it configured with the test samples), try:
```shell
node matcher.js "Bob"
```
which will produce:
```json
{
"name": "MinionIntent",
"slots": {
"MinionSlot": {
"name": "MinionSlot",
"value": "Bob"
}
}
}
```
or you could specify a particular recognizer file to use, e.g.:
```shell
node matcher.js "Bob" "./recognizer.json"
```
or try
```shell
node matcher.js "here is four hundred eighty eight million three hundred fifty two thousand five hundred twelve and also six oh three five five five one two one two"
```
which will produce:
```json
{
"name": "BlahIntent",
"slots": {
"BlahSlot": {
"name": "BlahSlot",
"value": "488352512"
},
"BlehSlot": {
"name": "BlehSlot",
"value": "6035551212"
}
}
}
```
```shell
node matcher.js "thirty five fifty one"
```
which will produce:
```json
{
"name": "FourDigitIntent",
"slots": {
"FooSlot": {
"name": "FooSlot",
"value": "3551"
}
}
}
```
```shell
node matcher.js "sure"
```
which will produce:
```json
{
"name": "AMAZON.YesIntent",
"slots": {}
}
```
```shell
node matcher.js "New England includes New Hampshire as one of its states"
```
which will produce:
```json
{
"name": "StateIntent",
"slots": {
"StateSlot": {
"name": "StateSlot",
"value": "New Hampshire"
}
}
}
```
```shell
node matcher.js "My first name is Jim"
```
which will produce:
```json
{
"name": "FirstNameIntent",
"slots": {
"FirstNameSlot": {
"name": "FirstNameSlot",
"value": "Jim"
}
}
}
```
```shell
node matcher.js "December thirty first nineteen ninety nine"
```
which will produce:
```json
{
"name": "DateIntent",
"slots": {
"DateSlot": {
"name": "DateSlot",
"value": "1999-12-31"
}
}
}
```
```shell
node matcher.js "lets do it on tuesday"
```
which will produce:
```json
{
"name": "DayOfWeekIntent",
"slots": {
"DayOfWeekSlot": {
"name": "DayOfWeekSlot",
"value": "tuesday"
}
}
}
```
Please note that matcher.js is just a convenience and also serves as an example.
You will NOT be using it at run time (most likely, though some might find the use
for it).
If you are porting an existing Alexa skill (to, for example, Cortana),
you will probably deploy your code (that uses the parser) to some middleware layer, like this:
```
Alexa --------------> Alexa <-- middleware <---- Cortana
Skill <-------------- AWS Lambda --> AWS Lambda ----> Skill
```
Where in the middleware AWS Lambda (exposed via API Gateway) you will be able to
see the raw user text from Cortana (passed as the "message" field in the request),
then call it same way matcher.js does, get the resulting json and update the
intent in the request from "None" to the resulting json. The backend Lambda can
then process it further.
If you are using vui-ad-hoc-alexa-recognizer for new development you have two main options:
* use it just for NLP to map utterances to intents, which you will then use for branching code
* use higher level domain functionality to define much of your app/skill
### Matched custom slot values
There are differences between what kind of values different services may send.
Alexa appears to respect capitalization of the custom slot values (or it sends
lower case versions) while Cortana capitalizes the first letter under some
circumstances, but not always, and also adds a period at the end of utterances
or even other punctuation signs (I've seen entering a zip code changed from
"12345" to "Is 12345 ?"). To keep Cortana (as well as other "misbehaving" services)
behaving consistently, the returned matches use the capitalization of the custom
slot values supplied in the config file rather than what Cortana will send.
Thus, if your custom slot value (in the config file) is "petunia" then "petunia"
will be returned even if Cortana will send you "Petunia".
### Slot flags
In some cases you would like to match on a particular slot differently from the
standard algorithm. For example, if you are trying to get the user's first name
you may way to match on ANYTHING the user says so that unusual names are matched.
In this case you can modify your utterances file to include special flags, e.g.:
```
FirstNameIntent My first name is {FirstNameSlot: INCLUDE_WILDCARD_MATCH, EXCLUDE_VALUES_MATCH }
```
These flags will be used in parsing. Here are the different currently available
flags:
1. "INCLUDE_VALUES_MATCH", "EXCLUDE_VALUES_MATCH" - to include/exclude custom
slot values in the matching pattern.
2. "INCLUDE_WILDCARD_MATCH", "EXCLUDE_WILDCARD_MATCH" - to include/exclude
a wildcard in the matching pattern.
3. "SOUNDEX_MATCH" - to use SoundEx for matching (will match on values that sound like desired values, but are not necessarily spelled the same)
4. "INCLUDE_SYNONYMS_MATCH", "EXCLUDE_SYNONYMS_MATCH" - to include/exclude
synonym values in the matching pattern. This is only relevant for custom slot types that actually use synonyms.
5. "EXCLUDE_YEAR_ONLY_DATES" - this flag is only applied to the AMAZON.DATE type
slot and turns off parsing of a single number as a year. This is useful when
there are otherwise identical utterances that may match on a number or on a date. If the year only
match is allowed then there is no way to differentiate between the two.
6. "EXCLUDE_NON_STATES" - this flag is only applied to the AMAZON.US_STATE type
slot and turns off parsing of US territories and D.C.
7. "STATE" - this is a parametrized flag (see below). currently
it only applies to the AMAZON.Airport slot type and it restricts the matches to
the specified states.
8. "COUNTRY" - this is a parametrized flag (see below). currently
it only applies to the AMAZON.Airline and AMAZON.Airport slot types and it restricts the matches to
the specified countries.
9. "CONTINENT", "TYPE" - these are parametrized flags (see below). currently
they only apply to the AMAZON.Airline slot type and they restrict the matches to
the specified types and continents.
10. "SPORT", "LEAGUE" - these are parametrized flags (see below). currently
they only apply to the AMAZON.SportsTeam slot type and they restrict the matches to
the specified sports and leagues.
11. "INCLUDE_PRIOR_NAMES", "EXCLUDE_PRIOR_NAMES" - Currently these only apply to the AMAZON.SportsTeam and AMAZON.Corporation
slot type and they include/exclude the prior team (or corporation) names in the search. Default is EXCLUDE_PRIOR_NAMES.
If you don't specify any of these, then
INCLUDE_VALUES_MATCH and EXCLUDE_WILDCARD_MATCH will be used as the default. Also,
if you include by mistake both INCLUDE... and EXCLUDE... for the same flag, the
default value is (silently) going to be used. If you are concerned you can look
at the generated recognizer.json to see how the flags were parsed.
Also note that SOUNDEX_MATCH will automatically imply EXCLUDE_VALUES_MATCH and
EXCLUDE_WILDCARD_MATCH flags; however SOUNDEX_MATCH is only available for the
custom slot types at this time.
It would typically not be useful (at this time with only these sets of flags)
to specify INCLUDE... for both or EXCLUDE... for both wildcard and value matches
unless you are specify SOUNDEX_MATCH. If you are going to include
wildcard then there is no reason to include values as well - it will only slow
down the parsing. If you exclude both then it will be the same as if you had removed that slot
from the utterance completely. For this reason, parsing ignores these combinations.
If you specify INCLUDE_WILDCARD_MATCH then only the wild card will be used.
If you specify both EXCLUDE_VALUES_MATCH and EXCLUDE_WILDCARD_MATCH then only
EXCLUDE_WILDCARD_MATCH is used.
Also note that you have to be very careful when using wildcards. For example,
imagine this utterance instead of the above example:
```
FirstNameIntent {FirstNameSlot:INCLUDE_WILDCARD_MATCH,EXCLUDE_VALUES_MATCH}
```
This will match on ANYTHING the user says. So, DON'T use wildcards in "naked"
utterances (ones that use nothing but slots) unless you are absolutely SURE that
that is what you want. This is why these flags exist at the utterance level rather
than intent level.
Also, you should probably NOT specify wildcard matches on slots of many of the
built in intents, such as date or number - this will likely not end well and
it doesn't make sense. For this reason, parsing ignores these flags at this
time on most of these slot types. Parsing will also ignore SOUNDEX_MATCH on non-custom
slot types (though this may be added in the future for some built in types).
In the future there will be other flags added, possibly specific to
particular built in slot types (e.g. I may add a flag to return only the female
first names from the AMAZON.US_FIRST_NAME type slot or numeric values within
a certain range from the AMAZON.NUMBER type slot).
### Parameterized flags
Some flags can take parameters. For example, "COUNTRY", "CONTINENT", and "TYPE" flags
are used to specify countries, continents, and types to use in the match. For example,
if your utterances file contains these lines:
```shell
AirlineIntent {AirlineSlot:COUNTRY(["canada"])} is a canadian airline
AirlineIntent {AirlineSlot:CONTINENT(["north america"])} is a north american airline
```
then only Canadian airlines will match the first one, and only north american
airlines will match the second one.
### Custom list based slot types with synonyms
You can create a very simple custom slot type based on a list of simple values, loaded either from config.json or a separate
text file. But you can also load "values" which are objects. These objects must themselves contain a "value" field
that replaces the simple field. In addition, these objects can also contain a field "synonyms" which must be an array
of strings. For example, here is a custom slot type defined in a config.json:
```json
{
"name": "KITCHENSTUFF",
"values": [
"Spoon",
{
"value": "Pan",
"synonyms": [
"Skillet"
]
}
]
}
```
This will match on "spoon", "pan", and "skillet". Furthermore, **and this is the real value of the synonyms**, when
matching on the "skillet", the actual returned value will be "Pan" (otherwise you could have simply added more values
instead of using synonyms).
Couple of important points:
* You can mix strings and objects within config.json
* If you want to specify json objects in a separate file then you MUST use a file with a .json extension and it must
contain valid json
* Whatever you specify in a file that does NOT have .json extension will be loaded as plain strings (one per line) even
if it contains valid json
#### Synonyms and SOUNDEX
Custom slot type that has synonyms will work with SOUNDEX flag just like one without synonyms.
### Custom slot types based on regular expressions
In addition to the normal custom type slots - one based on a list of values - you can also define a custom slot type
based on a regular expression. This might be useful if you are looking for some value that has a particular format.
For example, a serial number for a product might have a specific format and you may be looking for it in user input.
It would be impractical to specify all the serial numbers even if you had the up to date list. Instead, you can define
a custom slot that will match the regular expression for the serial number and return it.
E.g. given config.json:
```text
...
{
"name": "CUSTOMREGEXP",
"customRegExpString": "(ABC123|XYZ789)"
}
...
```
and otherwise standard intents and utterances files, when
```shell
node matcher.js "here is XYZ789 if you see it"
```
will produce:
```json
{
"name": "CustomRegExpIntent",
"slots": {
"CustomRegExpSlot": {
"name": "CustomRegExpSlot",
"value": "XYZ789"
}
}
}
```
You can also load the reg ex for a custom slot from a file. This can be useful for sharing the same reg ex between many
different recognizers. To do this, use customRegExpFile member instead of customRegExpString:
```text
...
{
"name": "CUSTOMREGEXP",
"customRegExpFile": "customRegExpFile.txt"
}
...
```
### Options list
Instead of creating multiple similar utterance lines like you would do with Alexa
utterances, you can specify variations with options lists:
```text
DateIntent I {want to|wish to|like to|would like to|can} meet {you|with you} {DateSlot}
```
is equivalent to these:
```text
DateIntent I want to meet you {DateSlot}
DateIntent I want to meet with you {DateSlot}
DateIntent I wish to meet you {DateSlot}
DateIntent I wish to meet with you {DateSlot}
DateIntent I like to meet you {DateSlot}
DateIntent I like to meet with you {DateSlot}
DateIntent I would like to meet you {DateSlot}
DateIntent I would like to meet with you {DateSlot}
DateIntent I can meet you {DateSlot}
DateIntent I can meet with you {DateSlot}
```
Note that you can specify an options list that's optional by omitting one value:
```text
DateIntent I {want to|wish to|like to|would like to|can} meet {|you|with you} {DateSlot}
```
will match on (for instance):
```text
I want to meet tomorrow
```
### Text equivalents
Similarly to the options list, text equivalents allow variations in the text to be matched. Unlike the options list
you don't have to manually add all of the possibilities. Instead, simply enclose the text that should match using
equivalents and this module will do it for you, e.g.:
```text
HiIntent {~Hi what time is it}
```
is equivalent to these:
```text
HiIntent Hi what time is it
HiIntent Hello what time is it
HiIntent Hey what time is it
HiIntent How are you what time is it
HiIntent Good morning what time is it
HiIntent Good day what time is it
HiIntent Good night what time is it
HiIntent Hi there what time is it
...etc..
```
At this time the following implementation is in place: it uses two data sets - a very small default data set and a
common misspellings data set. It will match both single
word substitutions AND phrase substitutions. This will soon be expanded to include
an independent npm module containing additional values (so that they can be updated independently of this module) as well
as the ability to add your own modules to support special "domain" equivalents. For example, you can add "slang" data
set or "medical jargon" data set.
So, an example similar to the above that substitutes both the phrases and the individual words and uses multiple data
sets (e.g. correcting for typos, skipping optional words like please, etc) could be:
```text
HiTimeIntent {~How are you can you tell me please what is the acceptable time to come to work}
```
is equivalent to these:
```text
HiTimeIntent how are you can you tell me please what is the acceptable time to come to work
HiTimeIntent how are you doing can you tell me please what is the acceptable time to come to work
HiTimeIntent how are you can you tell me what is the acceptable time to come to work
HiTimeIntent how are you doing can you tell me what is the acceptable time to come to work
HiTimeIntent how are you can you tell me please what is the acceptible time to come to work
HiTimeIntent how are you doing can you tell me please what is the acceptible time to come to work
HiTimeIntent how are you can you tell me what is the acceptible time to come to work
HiTimeIntent how are you doing can you tell me what is the acceptible time to come to work
HiTimeIntent hi can you tell me please what is the acceptable time to come to work
HiTimeIntent hello can you tell me please what is the acceptable time to come to work
HiTimeIntent good morning can you tell me please what is the acceptable time to come to work
HiTimeIntent good day can you tell me please what is the acceptable time to come to work
HiTimeIntent good evening can you tell me please what is the acceptable time to come to work
HiTimeIntent good night can you tell me please what is the acceptable time to come to work
HiTimeIntent whats up can you tell me please what is the acceptable time to come to work
HiTimeIntent hey can you tell me please what is the acceptable time to come to work
HiTimeIntent hi can you tell me what is the acceptable time to come to work
HiTimeIntent hello can you tell me what is the acceptable time to come to work
HiTimeIntent good morning can you tell me what is the acceptable time to come to work
HiTimeIntent good day can you tell me what is the acceptable time to come to work
HiTimeIntent good evening can you tell me what is the acceptable time to come to work
HiTimeIntent good night can you tell me what is the acceptable time to come to work
HiTimeIntent whats up can you tell me what is the acceptable time to come to work
HiTimeIntent hey can you tell me what is the acceptable time to come to work
HiTimeIntent hi can you tell me please what is the acceptible time to come to work
HiTimeIntent hello can you tell me please what is the acceptible time to come to work
HiTimeIntent good morning can you tell me please what is the acceptible time to come to work
HiTimeIntent good day can you tell me please what is the acceptible time to come to work
HiTimeIntent good evening can you tell me please what is the acceptible time to come to work
HiTimeIntent good night can you tell me please what is the acceptible time to come to work
HiTimeIntent whats up can you tell me please what is the acceptible time to come to work
HiTimeIntent hey can you tell me please what is the acceptible time to come to work
HiTimeIntent hi can you tell me what is the acceptible time to come to work
HiTimeIntent hello can you tell me what is the acceptible time to come to work
HiTimeIntent good morning can you tell me what is the acceptible time to come to work
HiTimeIntent good day can you tell me what is the acceptible time to come to work
HiTimeIntent good evening can you tell me what is the acceptible time to come to work
HiTimeIntent good night can you tell me what is the acceptible time to come to work
HiTimeIntent whats up can you tell me what is the acceptible time to come to work
HiTimeIntent hey can you tell me what is the acceptible time to come to work
```
Note that the matching algorithm is pretty efficient and does NOT actually try to match on these utterances, but instead
uses a single regular expression.
### Removing flags/cleaning up the utterance file
There is also a utility available to "clean up" utterance files for use with
Alexa. This may be needed if you want to use a single file as your utterances
file for both Alexa and porting projects. Since the slot flags don't exist in
Alexa, they need to be stripped from the utterance file. For that use
alexifyutterances.js utility:
```shell
node alexifyutterances.js --utterances test/utterances.txt --intents test/intents.json --output testutterances.txt --noconfig
Result was saved to testutterances.txt
```
You can now import testutterances.txt into the Alexa developer console.
Note that not only will alexifyutterances.js remove flags, it will also "unfold"
options lists into multiple utterances as well as "unfold" any text equivalents so that you can use them with Alexa.
This feature would be useful even if you only want to use this module to
reduce the tedium of entering multiple lines into Alexa and don't even
intent to create your own chat bot or convert your Alexa skill.
There is also support for the BETA Amazon interaction model editor. You can edit the files it generates to add
features supported by this module. E.g. you can options lists or slot flags or even TRANSCEND native slot types.
Then run it through the alexifyutterances.js and the result will be importable back into Alexa console:
```shell
node alexifyutterances.js --interactionmodel test/interactionmodel.json --output alexifiedmodel.json --noconfig
Result was saved to alexifiedmodel.json
```
### Nominal support for some built in list slots
Many of the list slots (e.g. AMAZON.Actor) have very large value lists. These
are often not needed in a typical vui skill. Thus, a compromise support is
provided for them. They are there and can be used, but they only have a few
values. If you actually do have a need for them, you have two options:
1. You can provide your own expansion list of values in the config.json file
2. You can use wildcard slot matching to match on any value the user can provide
### Transform functions
#### Custom transform functions
You can transform matched values before returning them. You do this by specifying
transform functions in the config file, here are examples for the built in and
custom slot types:
```json
{
"customSlotTypes":[
{
"name": "SOME",
"values": [
"apple",
"star fruit",
"pear",
"orange"
],
"transformSrcFilename": "./test/transformSome.js"
}
],
"builtInSlots": [
{
"name": "AMAZON.US_STATE",
"transformSrcFilename": "./test/transformUsState.js"
},
{
"name": "AMAZON.Month",
"transformSrcFilename": "./test/transformFirstWordTitleCase.js.js"
}
]
}
```
You then put into the specified "transformSrcFilename" file the source code for
the function to do the transformation.
Then, when you type:
```shell
node matcher.js 'january is the best month'
```
you will get (note the capitalized first letter of the month):
```json
{
"name": "MonthIntent",
"slots": {
"MonthSlot": {
"name": "MonthSlot",
"value": "January"
}
}
}
```
See the test directory for more examples.
There are many reasons you may want to do this: transforming states into postal
code or fixing issues with speech recognition, etc.
For example, a particular service may not understand some spoken phrases well.
One that I've ran into is the word "deductible" is understood to be "the duck tibble".
This will never match. Well, you could add this to your list of acceptable values.
This will only solve half a problem. Once you match it and send it to your Alexa
backend, it will choke on this. So, you can add a transform function to map
"the duck tibble" to "deductible" before sending it off to Alexa backend.
When you write a custom transform function be aware that it has this signature:
```javascript
function someTransformFunction(value, intentName, slotName, slotType){/* do something and return transformed value*/};
```
And that it returns a transformed value (or undefined if the input value is undefined or null).
You can use the other arguments to change how your function may transform the matched value.
For example, you may specify a particular transform function of a slot type, but you may check within your function
that the slot name equals a particular slot name and change the transformation.
#### Built in transform functions
In addition to being able to write your own custom transform functions you can also use some built in ones.
The current list is:
```text
addAngleBrackets - surrounds the matched value with <>
addCurlyBrackets - surrounds the matched value with {}
addParentheses - surrounds the matched value with ()
addSquareBrackets - surrounds the matched value with []
codeToState - converts passed in US state postal code to the corresponding state name. Does NOT convert territories.
formatAsUsPhoneNumber1 - formats TRANSCEND.US_PHONE_NUMBER matched value as (111) 111-1111
formatAsUsPhoneNumber2 - formats TRANSCEND.US_PHONE_NUMBER matched value as 111.111.1111
formatAsUsPhoneNumber3 - formats TRANSCEND.US_PHONE_NUMBER matched value as 111 111 1111
formatAsUsPhoneNumber4 - formats TRANSCEND.US_PHONE_NUMBER matched value as 111-111-1111
removeDigits - removes all digits from the matched value
removeDollar - removes all occurrences of $s from the matched value
removeNonDigits - removes all non-digits from the matched value
removeNonAlphanumericCharacters - removes all non alphanumeric characters from the matched value
removeNonWordCharacters - same as removeNonAlphanumericCharacters, but allows underscore. Removes anything that's not a number, letter, or underscore from the matched value.
removePeriod - removes all occurrences of . from the matched value
removePoundSign - removes all occurrences of #s from the matched value
removeWhiteSpaces - removes all continuous sequences of any white space characters from the matched value
replaceWhiteSpacesWithSpace - replaces all continuous sequences of any white space characters in the matched value with a single space
stateToCode - converts passed in US state name to the corresponding postal code. Does NOT convert territories.
toLowerCase - converts the matched value to lower case
toUpperCase - converts the matched value to upper case
```
You can also
see the currently available ones in the builtintransforms directory.
To use them, specify "transformBuiltInName" member instead of the "transformSrcFilename":
```json
{
"name": "MEANINGLESS",
"values": [
"foo",
"bar"
],
"transformBuiltInName": "toUpperCase"
}
```
#### Chaining transform functions
Both custom and built in transform functions can be chained simply by specifying an array instead of a single value in
the configuration file, for example:
```json
{
"name": "MEANINGLESS",
"values": [
"foo",
"bar"
],
"transformBuiltInName": ["toUpperCase", "addParentheses", "addSquareBrackets"]
}
```
will apply all the specified transforms.
### Mix-ins
Sometimes you may want to do some additional processing of the result before returning it. It could be almost anything,
for example:
* add logging to all matches
* compute sentiment score and add it to the result
* adjust/update/replace matched slot values
and many other possible examples.
Mix-in (or add on) processing allows you to do that and you can do it mostly through configuration (some coding may be
required)
#### Built in mix-ins
Currently there are only about seven built in mix ins. Here is the list with a short description for each:
* adddefaultslots - can be used to inject slot(s) with hard coded values
* changeintent - can be used to change the matched intent to another one
* charactercount - counts the characters in the matched utterance and attaches this count to the result
* countregexp - counts the occurence of the specified reg exp and attaches this count to the result
* noop - a simple logging mix in. Does not modify the result in any way, simply logs it to console
* removeslots - removes all matched slots from the result
* wordcount counts the words in the matched utterance and attaches this count to the result
Imagine that you update you config.json file to add the mixIns section like this:
```text
"mixIns": {
"bundles": [
{
"bundleName": "loggingMixIn",
"mixInCode": [
{
"mixInBuiltInName": "noop",
"arguments": {
"log": true
}
}
]
}
],
"appliesTo": [
{
"bundleName": "loggingMixIn",
"intentMatchRegExString": "(.*)"
}
]
}
```
What this does is defines a mix in "bundle" (i.e. bundle of the code - noop - and argument) and give it a name "loggingMixIn".
Then it specifies that this "bundle" applies to every intent (i.e. "appliesTo" field has a pairing of this bundle with
the "intentMatchRegExString" which matches on every intent: (.*)). As a result, the "noop" mix in will run after every
match and log the results. You can modify which intents it applies to by chaining the matching reg exp. The code that
will actually be run is noop.js located in the builtinmixins directory:
```javascript
"use strict";
module.exports = function(standardArgs, customArgs){ // eslint-disable-line no-unused-vars
let intentName;
let utterance;
let priorResult;
if(typeof standardArgs !== "undefined"){
intentName = standardArgs.intentName;
utterance = standardArgs.utterance;
priorResult = standardArgs.priorResult;
}
if(typeof cus