In API 26 (Oreo) a new TextClassification system was introduced. This has been further refined in API 28 (Pie). In this short series we’ll take a look at what this is, how to use it, and how we can add custom behaviours to it.
TextClassification is a mechanism by which the system can identify a specific type of text and add appropriate actions when the user selects that text. Some common classification types are phone numbers, email addresses, and URLs, and these would trigger actions to launch the system dialer, email client, and web browser respectively. All of this is performed by the default TextClassification service which is built in to Android, so the first thing we’ll look at is precisely how this works.
We can get the default TextClassificationManager by retrieving the appropriate system service:
textClassificationManager = getSystemService(Context.TEXT_CLASSIFICATION_SERVICE) as TextClassificationManager
When manually performing Text Classification it is worth remembering that this may be a computationally expensive operation because the default system TextClassifier uses a Machine Learning model to perform the classification. For this reason I have wrapped all of these calls inside an async coroutine with CommonPool as the context, so it will effectively run on a background thread:
class MainActivity : AppCompatActivity() {
private val emailText = "[email protected]"
private val urlText = "https://blog.stylingandroid.com"
private val hybridText = "Email: $emailText"
private lateinit var textClassificationManager: TextClassificationManager
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
classifier()
}
private fun classifier() = async(CommonPool) {
textClassificationManager = getSystemService(Context.TEXT_CLASSIFICATION_SERVICE) as TextClassificationManager
...
}
}
There are two basic operations that comprise classifying text. The first is the classification itself. To do this we pass in a string containing the text that we wish to classify, the start and end of the subsection that will actually be classified, and a list of locales. The first three arguments are fairly self explanatory, but the final one requires a little explanation. Earlier I mentioned that the default system TextClassifier uses an ML model to perform the classification but it actually has multiple models for different languages and locales, and we need to specify the locales that we’re interested in so that it uses the correct model. It is worth keeping this list as small as possible because evaluating using multiple ML models is going to be more expensive computationally.
To perform the classification of an email address we first obtain a TextClassifier instance from TextClassificationManager and call its classifyText() method:
val textClassifier = textClassificationManager.textClassifier val emailClassification = textClassifier.classifyText(emailText, 0, emailText.length, LocaleList.getDefault()) println(emailClassification)
The TextClassification instance that is returned looks like this:
TextClassification {[email protected], entities={email=1.0}, actions=[android.app.RemoteAction@4e67771, android.app.RemoteAction@eb7956], id=androidtc|en_v6|754483982}
We can see that it identifies the text as an email address with a confidence level of 1.0 (which is on a scale of 0.0-1.0, so is a certain match). Even though this is clearly a dummy email address to the human eye, it still meets the criteria for being a valid email address.
We can perform another classification using the same TextClassifier instance, but with a string containing a URL instead:
val urlClassification = textClassifier.classifyText(urlText, 0, urlText.length, LocaleList.getDefault()) println(urlClassification)
This time the resulting TextClassification identifies this as a URL:
TextClassification {text=https://blog.stylingandroid.com, entities={url=1.0}, actions=[android.app.RemoteAction@33dd4e2], id=androidtc|en_v6|-1332134748}
As well as the identifying the specific types of text that were identified, the TextClassification also contains zero or more actions which can handle the type that was identified. This is encapsulated in a RemoteAction object which includes a PendingIntent. We can invoke the RemoteAction which will trigger the PendingIntent with a payload of the text. The RemoteAction returned when we detected an email address would trigger a PendingIntent to launch a mail client to compose an email to the email address, and the one returned for the URL will launch a web browser to view the URL. We’ll look more at RemoteActions later in the series.
One important thing to note here is that when we call classifyText() the start and end values must precisely delimit a substring which contains a given type of classification. In other words, if we use a string of “Email: [email protected]” then performing a classification of the entire string would not return an email address type, but ‘other’. It is only if we pass in the start and end character indicies which delimit the “[email protected]” substring that it will identify it as an email address.
This begs the question: How do we determine the start and end of substring which can be correctly classified? That is where the other operation of TextClassifier comes in. The suggestSelection() method identifies a substring which can be classified in to a concrete type, but it works slightly differently than one might imagine. If we look at the previous example of “Email: [email protected]” we might expect that if we pass in the entire string it will identify the correct substring, but that’s not how it works. Rather than narrowing the selection down from the entire string to a smaller substring, it actually grows a selection from a given substring of indeterminate type to a larger substring with a concrete type. The use-case here is when a user long presses on a TextView, the initial selection will be a single character, and TextClassifier can expand this selection. In U terms, this means that if a user long presses on a long string containing an email address, the initial selection will be very small, but will expand to the email address.
We can see this in action by calling suggestSelection() with the same set of arguments as classifyText(). In this case the start and end positions delimit a single character which appears within the email address section of the string:
val suggestions = textClassifier.suggestSelection(hybridText, 10, 11, LocaleList.getDefault()) println(suggestions)
This returns a TextSelection instance which contains both the start and end of the substring which encapsulates the detected email address, but also the type and confidence score:
TextSelection {id=androidtc|en_v6|-456509634, startIndex=7, endIndex=22, entities={email=1.0}}
We could now use the start and end values for a call to classifyText(), but in reality we don’t have to. The two main use-cases for using TextClassifier are TextView and WebView and they both already use it. Here we can see how long pressing on an email address or URL in a TextView which is selectable expands the selection, and provides a popup which performs an action specific to the type of text selected. In this case it opens the URL in Chrome:
I really can’t think of any use-cases where you may want to invoke TextClassifier directly unless you have a custom View which permits the selection of a block of text but extends neither TextView nor WebView. If this is the case then you have my condolences.
You may now be thinking that I have just wasted the time it has taken you to read this article, but I assure you that I have not. While actually invoking these APIs is something that very few Android developers will ever have to do, implementing custom classifiers is another thing altogether, and a good understanding of how TextClassifier works is a fundamental requirement to writing your own.
In the next article in this series, we’ll look at how we can do that.
The source code for this article is available here.
© 2018, Mark Allison. All rights reserved.
Copyright © 2018 Styling Android. All Rights Reserved.
Information about how to reuse or republish this work may be available at http://blog.stylingandroid.com/license-information.
