Highlighting The Text While Speech Is Progressing

I'm developing an App in which I've textview consists of String and two buttons. When I click the speak button, the text gets converted to speech. But I want to Highlight the word

Solution 1:

For Android API 26 and above AND a TTS engine that supports onRangeStart (in this case, Google TTS):

publicclassMainActivityextendsAppCompatActivityimplementsTextToSpeech.OnInitListener {

    TextToSpeech tts;

    String sentence = "The Quick Brown Fox Jumps Over The Lazy Dog.";

    TextView textView;

    @OverrideprotectedvoidonCreate(Bundle savedInstanceState) {

        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        textView = findViewById(R.id.textView);
        textView.setText(sentence);
        tts = newTextToSpeech(this, this);

    }

    // TextToSpeech.OnInitListener (for our purposes, the "main method" of this activity)publicvoidonInit(int status) {

        tts.setOnUtteranceProgressListener(newUtteranceProgressListener() {

            @OverridepublicvoidonStart(String utteranceId) {
                Log.i("XXX", "utterance started");
            }

            @OverridepublicvoidonDone(String utteranceId) {
                Log.i("XXX", "utterance done");
            }

            @OverridepublicvoidonError(String utteranceId) {
                Log.i("XXX", "utterance error");
            }

            @OverridepublicvoidonRangeStart(String utteranceId,
                                     final int start,
                                     final int end,
                                     int frame) {
                Log.i("XXX", "onRangeStart() ... utteranceId: " + utteranceId + ", start: " + start
                        + ", end: " + end + ", frame: " + frame);

                // onRangeStart (and all UtteranceProgressListener callbacks) do not run on main thread// ... so we explicitly manipulate views on the main thread:runOnUiThread(newRunnable() {
                    @Overridepublicvoidrun() {

                        Spannable textWithHighlights = newSpannableString(sentence);
                        textWithHighlights.setSpan(newForegroundColorSpan(Color.YELLOW), start, end, Spanned.SPAN_INCLUSIVE_INCLUSIVE);
                        textView.setText(textWithHighlights);

                    }
                });

            }

        });

    }

    publicvoidstartClicked(View ignored) {

        tts.speak(sentence, TextToSpeech.QUEUE_FLUSH, null, "doesn't matter yet");

    }

}

// -------------------------------------------------------------------

Android API 25 and below:

In theory, the most intuitive way of accomplish this would be to:

1) Break the string into pieces

2) Detect when each piece has been/is being spoken

3)Highlight that piece accordingly

However, unfortunately, when using the Android TextToSpeech class where the speech output is generated in real-time, the smallest unit of speech that you are able to precisely detect the progress of (using UtteranceProgressListener) is an utterance (whatever string you decided to send to the TTS) -- not necessarily a word.

There is no mechanism whereby you can simply send a multi-word string as an utterance, and then somehow detect exactly when each word has been spoken.

Therefore, in order to (easily) highlight each word in order, you would have to either:

A) Send each word to the TTS individually as a single utterance (but this will cause disjointed pronunciation), or

B) Highlight sentence-by-sentence instead, sending each sentence as an utterance (easiest method, but not your desired behaviour).

If you really insist on achieving a word-by-word highlighting effect, the only way I can think of (using Android TextToSpeech) is to use sentence-size utterances, but instead of using speak(), use synthesizeToFile()... and then use a media player or sound player of some sort to play the speech back... somehow approximating the timing of the highlights in terms of where the nth word lies relative to the total audio file length. So, for example, if the sentence is 10 words long, and the file is 30% complete, then you would highlight the 4th word. This would be difficult and inexact, but theoretically possible.

There are obviously apps and games that already exist that do this... games like Parappa the Rapper, or karaoke apps, but I think the way they do it is by having pre-recorded/static audio files with markers encoded at exact times that trigger the highlights. If your text content is always going to be the same, and only in one language, then you could also do this.

However, if the spoken text is user-entered or unknown until runtime, requiring a TTS, then I don't know of any straight-forward solution.

If you decide on one of these more narrowed-down approaches, then I would suggest posting a new question accordingly.

Solution 2:

If you want to change paragraph color which is current speaking in TTS.

This code works in Google TTS, Samsung TTS and also other TTS engines

First you have to implement TextToSpeech.OnInitListener Like (public class MainActivity extends AppCompatActivity implements TextToSpeech.OnInitListener)

This is all essential object and variable used in this code.

privateString sentance = "";
privateString typingString = "";
privateint paragraphCount = 0;
private HashMap<String, String> map = new HashMap<>();
private ArrayList<String> stringArrayList = new ArrayList<>();

In your activity onCrreat() method

tts = newTextToSpeech(this, this);
map = new HashMap<>();
map.put(TextToSpeech.Engine.KEY_PARAM_UTTERANCE_ID, "UniqueID")

Paste this method into your activity and call this method at the click of a button.

privatevoidnewPlayMethod() {
    if (paragraphCount == 0) {
        stringArrayList = new ArrayList<>(Arrays.asList("Your Document texts".split("\n")));
    }
    try {
        SpannableString spannableString = new SpannableString(tvData.getText().toString());
        spannableString.setSpan(new ForegroundColorSpan(getResources().getColor(R.color.colorPrimaryDark)),
                0, tvData.getText().toString().length(), 33);
        spannableString.setSpan(new ForegroundColorSpan(getResources().getColor(R.color.colorAccent)),
                tvData.getText().toString().indexOf(stringArrayList.get(paragraphCount)),
                tvData.getText().toString().indexOf(stringArrayList.get(paragraphCount)) +
                        stringArrayList.get(paragraphCount).length(),
                33);

        tts.speak(stringArrayList.get(paragraphCount), TextToSpeech.QUEUE_FLUSH, map);

        tvData.setText(spannableString);
    } catch (Exception e) {
        e.printStackTrace();
    }
}

Put below code into @Override onInit method.

 tts.setOnUtteranceProgressListener(newUtteranceProgressListener() {

        @OverridepublicvoidonStart(String utteranceId) {
            Log.i("TTS", "utterance started");
        }

        @OverridepublicvoidonDone(String utteranceId) {
            if (stringArrayList.size() != paragraphCount) {
                paragraphCount++;
                newPlayMethod();
            } else {
                paragraphCount = 0;
            }
            Log.i("TTS", "utterance done");
        }

        @OverridepublicvoidonError(String utteranceId) {
            Log.i("TTS", "utterance error");
        }

    });