Diederik Krols

The XAML Brewer

A Speech Dialog Box for Universal Windows Phone apps

In this article I present a Universal Control to assist in text-to-speech and speech-to-text use cases on Windows Phone 8.1. The Speech Dialog Box is a templatable custom control in a shared universal project. Its default look is inspired by the Cortana UI: a text block with a microphone button attached to it:

Cortana SDB_Default

The Speech Input Dialog is an evolution of the Speech Input Box that I used in the previous two blog posts. This newer version of the control has a more ambitious purpose: it is designed to not only recognize and repeat speech input, but to engage a full conversation. Here’s a class diagram:

Style

The Speech Dialog Box comes as a localizable and templatable Custom Control. Its default style lives in the generic.xaml file. This style embeds a MediaElement, which is necessary for speaking, and a different grid for each relevant state. The control’s states are

  • Default: the control displays its current content (Question or Text) in a TextBlock, and also a Button to start listening.
  • Listening: the control shows in a TextBlock that he’s listening to you, and also a Button to cancel.
  • Typing: the control accepts typed input through a TextBox.
  • Thinking: the control shows in a TextBlock that he’s trying to recognize the input, and also a Button to cancel.
  • The Speaking state has no particular UI.

You don’t need to override this template to make the control blend into your design. The following properties are available to tweak its look:

  • Foreground: the color of the text in default and typing modes, becomes background color in other modes
  • Background: the background color in default and typing modes
  • Highlight: the secondary color, used for the border in typing mode and for the text and icons in listening mode
  • ButtonBackground: the background color for the button in default mode

Here’s and overview of the control in the different states, using the default color scheme:

SDB_Question SDB_Typing SDB_Listening SDB_Thinking

Managing a Conversation

The Speech Dialog Box comes with the following public members that help you to set up a two-way conversation:

  • VoiceGender: sets the gender to use for the voice (note: the language is taken from the UI culture).
  • Question: sets the question that the control will ask you.
  • Constraints: the list of constraints for speech recognition, e.g. a Semantic Interpretation for Speech Recognition (SISR) – there’s a nice example of this here.
  • StartListening: sets the control to listening mode.
  • Text: the recognized text.
  • TextChanged: this event is raised when the control has finished the text recognition.
  • ResponsePattern: the string format that specifies how the control will reply the recognized text to you, e.g. “I understood {0}”.
  • Speak: lets the control repeat the text.
  • Speak(string text): lets the control speak the specified text in its current voice.
  • SpeakSsml(string ssml): lets the  control speak the specified Speech Synthesis Markup Language (SSML) text in its current voice.

An example

The sample app contains buttons that demonstrate all of the control’s features. Here’s how the Speech Dialog Box is defined in XAML:

<controls:SpeechDialogBox x:Name="SpeechDialogBox"
                          Background="White"
                          Foreground="Black"
                          ButtonBackground="DimGray"
                          Highlight="DarkOrange" />

Here’s the code behind the ‘conversation’ button:

private async void ConversationButton_Click(object sender, RoutedEventArgs e)
{
    // Set the question.
    this.SpeechDialogBox.Question = "What's your favorite color?";

    // Let the control ask the question out loud.
    await this.SpeechDialogBox.Speak("What is your favorite color?");

    // Reset the control when it answered (optional).
    this.SpeechDialogBox.TextChanged += this.SpeechInputBox_TextChanged;

    // Teach the control to recognize the colors of the rainbow in a random text.
    var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets//ColorRecognizer.xml"));
    var grammarFileConstraint = new SpeechRecognitionGrammarFileConstraint(storageFile, "colors");
    this.SpeechDialogBox.Constraints.Clear();
    this.SpeechDialogBox.Constraints.Add(grammarFileConstraint);
            
    // Format the spoken response.
    this.SpeechDialogBox.ResponsePattern = "What a coincidence. {0} is my favorite color too.";
            
    // Start listening
    this.SpeechDialogBox.StartListening();
}

private async void SpeechInputBox_TextChanged(object sender, EventArgs e)
{
    this.SpeechDialogBox.TextChanged -= this.SpeechInputBox_TextChanged;
    await this.SpeechDialogBox.Reset();
}

Here’s how the conversation may look like:

  • Phone: “What is your favorite color?”
  • Person: “I think it’s blue today.”
  • Phone: “What a coincidence: blue is my favorite color too.”

At this moment (at least before the Reset) the Text property of the control has the value “blue”, so you can continue the conversation with it. Cool, isn’t it?

Source Code

After many years, I decided to stop attaching ZIP files to my blog posts. All the newer stuff will be shared –and updated- through GitHub. This solution was created with Visual Studio 2013 Update 4.

Enjoy!

Xaml Brewer

Integrating Cortana in your Universal Windows Phone app

This article describes how to register and use voice commands to start your Universal Windows Phone app, and how to continue the conversation within your app. The attached sample app comes with the following features:

  • It registers a set of voice commands with Cortana,
  • it recognizes a simple ‘open the door’ command,
  • it discovers whether it was started by voice or by text,
  • it recognizes a natural ‘take over‘ command, whit lots of optional terms,
  • it recognizes a complex ‘close a colored something’, where ‘something’ and ‘color’ come from a predefined list,
  • it modifies one of these lists programmatically,
  • it requests the missing color when you ask it to ‘close something’, and
  • it comes with an improved version of the Cortana-like SpeechInputBox control.

Here are some screenshots of this app. It introduces my new personal assistant, named Kwebbel (or in English ‘Kwebble’):

Real_Kwebbel takeover_text closedoor

Kwebble is written as a Universal app, I dropped the Windows app project because Cortana is not yet available on that platform. I you prefer to stick to Silverlight, check the MSDN Voice Search sample.

Registering the Voice Command Definitions

Your app can get activated by voice only after it registered its call sign and its list of commands with Cortana. This is done through an XML file with Voice Command Definitions (VCD). It contains different command sets – one for each language that you want to support. This is how such a command set starts: with the command prefix (“Kwebble”) and the sample text that will appear in the Cortana overview:

<!-- Be sure to use the v1.1 namespace to utilize the PhraseTopic feature -->
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.1">
  <!-- The CommandSet Name is used to programmatically access the CommandSet -->
  <CommandSet xml:lang="en-us" Name="englishCommands">
    <!-- The CommandPrefix provides an alternative to your full app name for invocation -->
    <CommandPrefix> Kwebble </CommandPrefix>
    <!-- The CommandSet Example appears in the global help alongside your app name -->
    <Example> Close the door. </Example>

Later in this article, we cover the rest of the elements. You don’t have to write this from scratch, there’s a Visual Studio menu to add a VCD file to your project:

AddVCDToProject

Here’s how to register the file through the VoiceCommandManager. I factored out all Cortana-related code in a static SpeechActivation class:

/// <summary>
/// Register the VCD with Cortana.
/// </summary>
public static async void RegisterCommands()
{
    var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets//VoiceCommandDefinition.xml"));
    await VoiceCommandManager.InstallCommandSetsFromStorageFileAsync(storageFile);
}

If your commands are registered and you ask Cortana “What can I say”, then the app is listed and the sample command is shown:

Cortana_1

Universal apps cannot –yet- define the icon to be displayed in that list.

A simple command

Let’s take a look at the individual commands. Each command comes with

  • Name: a technical name that you can use in your code,
  • Example: an example that is displayed in the Cortana UI,
  • One or more ListenFor elements: the text to listen for, the feedback from Cortana when she recognized the command, and
  • Navigate: the page to navigate to when the app is activated through the command.

The  Navigate element is required by the XSD, but it is used by Silverlight only: it is ignored by Universal apps.

Here’s an example of a very basic ‘open the door’ command, it’s just an exhaustive enumeration of the alternatives.

<Command Name="DoorOpen">
  <!-- The Command example appears in the drill-down help page for your app -->
  <Example> Door 'Open' </Example>

  <!-- ListenFor elements provide ways to say the command. -->
  <ListenFor> Door open </ListenFor>
  <ListenFor> Open door </ListenFor>
  <ListenFor> Open the door </ListenFor>

  <!--Feedback provides the displayed and spoken text when your command is triggered -->
  <Feedback> Opening the door ... </Feedback>

  <!-- Navigate specifies the desired page or invocation destination for the Command-->
  <!-- Silverlight only, WinRT and Universal apps deal with this themselves. -->
  <!-- But it's mandatory according to the XSD. -->
  <Navigate Target="OtherPage.xaml" />
</Command>

Here you see the list of commands in the Cortana UI, and the feedback:

Cortana_2 Cortana_3  

Since ‘Kwebble’ is not really an English word, Cortana has a problem recognizing it. I’ve seen the term being resolved into ‘Grebel’ (as in the screenshot), ‘Pueblo’, ‘Devil’ and very often ‘Google’. But anyway, ‘something that sounds like ‘Kwebble’ followed by ‘open the door’ starts the app appropriately. Strangely enough that’s not what happens with text input. If I type ‘Kwebbel close the door’ –another valid command- the app’s identifier is not recognized and I’m redirected to Bing:

textinput1 textinput2  

How the app reacts

A Universal app can determine if it is activated by Cortana in the OnActivated event of its root App class. If the provided event argument is of the type VoiceCommandActivatedEventArgs, then Cortana is responsible for the launch:

protected override void OnActivated(IActivatedEventArgs args)
{
    var rootFrame = EnsureRootFrame();

    // ...
           
    base.OnActivated(args);

#if WINDOWS_PHONE_APP
    Services.SpeechActivation.HandleCommands(args, rootFrame);
#endif

    // Ensure the current window is active
    Window.Current.Activate();
}
/// <summary>
/// Verify whether the app was activated by voice command, and deal with it.
/// </summary>
public static void HandleCommands(IActivatedEventArgs args, Frame rootFrame)
{
    if (args.Kind == ActivationKind.VoiceCommand)
    {
        VoiceCommandActivatedEventArgs voiceArgs = (VoiceCommandActivatedEventArgs)args;

        // ...

    }
}

The Result property of this VoiceActivatedEventArgs is a SpeechRecognitionResult instance that contains detailed information about the command. In Silverlight there’s a RuleName property that references the command. In Universal apps this is not there. I was first tempted to parse the full Text to figure out what command was spoken or typed, but that would become rather complex for the more natural commands. It’s easier and safer to walk through the RulePath elements – the list of rule identifiers that triggered the command. Here’s another code snippet from the sample app, the ‘Take over’ command guides us to the main page, the other commands bring us to the OtherPage. We conveniently pass the event arguments to the page we’re navigating to, so it can also access the voice command:

// First attempt:
// if (voiceArgs.Result.Text.Contains("take over"))
// Better:
if (voiceArgs.Result.RulePath.ToList().Contains("TakeOver"))
{
    rootFrame.Navigate(typeof(MainPage), voiceArgs);
}
else
{
    rootFrame.Navigate(typeof(OtherPage), voiceArgs);
}

Optional words in command phrases

The ListenFor elements in the command phrases may contain optional words. These are wrapped in square brackets. Here’s the TakeOver command from the sample app. It recognizes different natural forms of the ‘take over’ command, like ‘would you please take over’ and ‘take over my session please’:

<Command Name="TakeOver">
  <!-- The Command example appears in the drill-down help page for your app -->
  <Example> Take over </Example>

  <!-- ListenFor elements provide ways to say the command, including [optional] words -->
  <ListenFor> [would] [you] [please] take over [the] [my] [session] [please]</ListenFor>

  <!--Feedback provides the displayed and spoken text when your command is triggered -->
  <Feedback> Thanks, taking over the session ... </Feedback>

  <!-- Navigate specifies the desired page or invocation destination for the Command-->
  <!-- Silverlight only, WinRT and Universal apps deal with this themselves. -->
  <!-- But it's mandatory according to the XSD. -->
  <Navigate Target="MainPage.xaml" />
</Command>

When the command is fired, Kwebble literally takes over and starts talking. Since the VoiceCommandActivated event argument was passed from the app to the page, the page can further analyze it to adapt its behavior:

/// <summary>
/// Invoked when this page is about to be displayed in a Frame.
/// </summary>
protected async override void OnNavigatedTo(NavigationEventArgs e)
{
    if (e.Parameter is VoiceCommandActivatedEventArgs)
    {
        var args = e.Parameter as VoiceCommandActivatedEventArgs;
        var speechRecognitionResult = args.Result;

        // Get the whole command phrase.
        this.InfoText.Text = "'" + speechRecognitionResult.Text + "'";

        // Find the command.
        foreach (var item in speechRecognitionResult.RulePath)
        {
            this.InfoText.Text += ("\n\nRule: " + item);
        }

        // ...

        if (speechRecognitionResult.RulePath.ToList().Contains("TakeOver"))
        {
            await this.Dispatcher.RunAsync(
                Windows.UI.Core.CoreDispatcherPriority.Normal, 
                () => Session_Taken_Over(speechRecognitionResult.CommandMode()));
        }
    }
}

Detecting the command mode

If the app was started through a spoken Cortana command, it may start talking. If the command was provided in quiet mode -typed in the input box through the keyboard- then the app should also react quietly. You can figure out the command mode –Voice or Text- by looking it up in the Properties of the SemanticInterpretation of the speech recognition result. Here’s a static method that returns the command mode for a speech recognition result:

/// <summary>
/// Returns how the app was voice activated: Voice or Text.
/// </summary>
public static CommandModes CommandMode(this SpeechRecognitionResult speechRecognitionResult)
{
    var semanticProperties = speechRecognitionResult.SemanticInterpretation.Properties;

    if (semanticProperties.ContainsKey("commandMode"))
    {
        return (semanticProperties["commandMode"][0] == "voice" ? CommandModes.Voice : CommandModes.Text);
    }

    return CommandModes.Text;
}
    
/// <summary>
/// Voice Command Activation Modes: speech or text input.
/// </summary>
public enum CommandModes
{
Voice,
Text
}

Here’s how the sample app reacts to the ‘take over’ command. In Voice mode it loads an SSML document and starts talking, in Text mode it just updates the screen:

if (mode == CommandModes.Voice)
{
    // Get the prepared text.
    var folder = Windows.ApplicationModel.Package.Current.InstalledLocation;
    folder = await folder.GetFolderAsync("Assets");
    var file = await folder.GetFileAsync("SSML_Session.xml");
    var ssml = await Windows.Storage.FileIO.ReadTextAsync(file);

    // Say it.
    var voice = new Voice(this.MediaElement);
    voice.SaySSML(ssml);
}
else
{
// Only update the UI. this.InfoText.Text += "\n\nBla bla bla ...."; }

Here’s a screenshot of the app in both modes (just images, no sound effects):

takeover_voice takeover_text  

Natural command phrases

Do not assume that the user will try to launch your app by just saying the call sign and a command name: “computer, start simulation” is so eighties. Modern speech recognition API’s can deal very well with natural language and Cortana is no exception. The Voice Definition Command file can have more than just fixed and optional words (square brackets), it can also deal with so-called phrase lists and phrase topics. These are surrounded with curly brackets. The Kwebble app uses a couple of phrase lists. For an example of a phrase topic, check the already mentioned MSDN Voice Commands Quick start. The following command recognizes the ‘close’ action, followed by a color from a list, followed by the thing to be closed, also from a list. This ‘close’ command will be triggered by phrases like ‘close the door’, ‘close the red door’, and ‘close the yellow window’.

<Command Name="Close">
  <!-- The Command example appears in the drill-down help page for your app -->
  <Example> Close door </Example>

  <!-- ListenFor elements provide ways to say the command, including references to 
        {PhraseLists} and {PhraseTopics} as well as [optional] words -->
  <ListenFor> Close {colors} {closables} </ListenFor>
  <ListenFor> Close the {colors} {closables} </ListenFor>
  <ListenFor> Close your {colors} {closables} </ListenFor>

  <!--Feedback provides the displayed and spoken text when your command is triggered -->
  <Feedback> Closing {closables} </Feedback>

  <!-- Navigate specifies the desired page or invocation destination for the Command-->
  <!-- Silverlight only, WinRT and Universal apps deal with this themselves. -->
  <!-- But it's mandatory according to the XSD. -->
  <Navigate Target="MainPage.xaml" />
</Command>

<PhraseList Label="colors">
  <Item> yellow </Item>
  <Item> green </Item>
  <Item> red </Item>
  <!-- Fake item to make the color optional -->
  <Item> a </Item>
</PhraseList>

<PhraseList Label="closables">
  <Item> door </Item>
  <Item> window </Item>
  <Item> mouth </Item>
</PhraseList>

You can add optional terms to the ListenFor elements so that sentences like ‘Would you be so kind to close your mouth, please?’ would also trigger the close command. What you can not do, is define a phrase list as optional. Square brackets and curly brackets cannot surround the same term. As a workaround I added a dummy color called ‘a’. The fuzzy recognition logic will map ‘close door’ to ‘close a door’ and put ‘a’ and ‘door’ in the semantic properties of the speech recognition result. Here’s how the sample app evaluates these properties to figure out how to proceed:

var semanticProperties = speechRecognitionResult.SemanticInterpretation.Properties;

// Figure out the color.
if (semanticProperties.ContainsKey("colors"))
{
    this.InfoText.Text += (string.Format("\nColor: {0}", semanticProperties["colors"][0]));
}

// Figure out the closable.
if (semanticProperties.ContainsKey("closables"))
{
    this.InfoText.Text += (string.Format("\nClosable: {0}", semanticProperties["closables"][0]));
}

Continuing the conversation inside the app

Cortana’s responsibilities stop when it started up your app via a spoken or typed command. If you want to continue the conversation (e.g. for asking more details) then you have to do this inside your app. When the Kwebble sample app is started with a ‘close the door’ command without a color, then she will request for the missing color and evaluate your answer. Here’s how she detects the command with the missing color (remember: ‘a’ is the missing color):

// Ask for the missing color, when closing the door.
if (speechRecognitionResult.RulePath[0] == "Close" &&
    semanticProperties["closables"][0] == "door" &&
    semanticProperties["colors"][0] == "a")
{
    // ...

    if (speechRecognitionResult.CommandMode() == CommandModes.Voice)
    {
        var voice = new Voice(this.MediaElement);
        voice.Say("Which door do you want me to close?");
        voice.Speaking_Completed += Voice_Speaking_Completed;
    }
}

The Kwebble app comes with a Speech Input Box control, an improved version from the one I introduced in my previous blog post. I exposes its Text and its Constraints collection so you can change these. You can now fluently continue the conversation by triggering the ‘listening’ mode programmatically (skipping an obsolete Tap). And there’s more: I added the Cortana sound effect when the control starts listening.

SpeechInputBox2

Here’s what happens just before Kwebble ask you which door to close. The speech input box is prepared to recognize a specific answer to the question:

var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets//SpeechRecognitionGrammar.xml"));
var grammarFileConstraint = new Windows.Media.SpeechRecognition.SpeechRecognitionGrammarFileConstraint(storageFile, "colors");
this.SpeechInputBox.Constraints.Clear();
this.SpeechInputBox.Constraints.Add(grammarFileConstraint);
this.SpeechInputBox.Question = "Which door?";

She only starts listening after she finished asking the question. Otherwise she starts listening to herself. Seriously, I am not kidding!

private void Voice_Speaking_Completed(object sender, EventArgs e)
{
    this.SpeechInputBox.StartListening();
}

Natural phrases in the conversation

I already mentioned that the Voice Command Definitions for Cortana activation are quite capable of dealing with natural language, When you take over the conversation in your app, it gets even better. Using a SpeechRecognitionFileConstraint you can explain the SpeechInputBox (and the embedded SpeechRecognizer) the specific pattern to listen for, in Speech Recognition Grammar Specification (SRGS) XML format. When Kwebble asks you which door to close, she’s only interested in phrases that contain one of the door colors (red, green, yellow). Here’s the SRGS grammar that recognizes these, it just listens for color names, and ignores all the rest:

<grammar xml:lang="en-US"
         root="colorChooser"
         tag-format="semantics/1.0"
         version="1.0"
         xmlns="http://www.w3.org/2001/06/grammar">

  <!-- The following rule recognizes any phrase with a color. -->
  <!-- It's defined as the root rule of the grammar. -->
  <rule id="colorChooser">
    <ruleref special="GARBAGE"/>
    <ruleref uri="#color"/>
    <ruleref special="GARBAGE"/>
  </rule>

  <!-- The list of colors that are recognized. -->
  <rule id="color">
    <one-of>
      <item>
        red <tag> out="red"; </tag>
      </item>
      <item>
        green <tag> out="green"; </tag>
      </item>
      <item>
        yellow <tag> out="yellow"; </tag>
      </item>
    </one-of>
  </rule>

</grammar>

Here are some screenshots from the conversation. Kwebbel recognizes the missing color, then asks for it and start listening for the answer; then she recognizes the color:

closedoor Kwebbel_Listening reddoor

 

That’s all folks!

Here’s the full code of the sample app: the XAML, the C#, and last but not least the different powerful XML files. The solution was created with Visual Studio 2013 Update 3: U2UC.WinUni.Cortana.zip (164.6KB)

Enjoy!

XAML Brewer

Speech Recognition and Speech Synthesis in Universal Windows Apps

This article introduces you to speech recognition (speech-to-text, STT) and speech synthesis (text-to-speech, TTS) in a Universal Windows XAML app. It comes with a sample app – code attached at the end- that has buttons for

  • opening the standard UI for speech recognition,
  • opening a custom UI for speech recognition,
  • speaking a text in a system-provided voice of your choice, and
  • speaking a text from a SSML document.

On top of that, the Windows Phone version of the sample app comes with a custom control for speech and keyboard input, based on the Cortana look-and-feel.

Here are screenshots from the Windows and Phone versions of the sample app:

SpeechScreenshots

Speech Recognition

For speech recognition there’s a huge difference between a Universal Windows app and a Universal Windows Phone app: the latter has everything built in, the former requires some extra downloading, installation, and registration. But after that, the experience is more or less the same.

Windows Phone

For speech-to-text, Windows Phone comes with the SpeechRecognizer class, just create an instance of it: it will listen to you, think for a while, and then come up with a text string. You can make the recognition easier by giving the control some context in the form of Constraints, like

  • a predefined or custom Grammar – words and phrases that the control understands, or
  • a topic, or
  • just a list of words.

Contradictory to the documentation, you *have* to provide constraints, and a call to CompileConstraintsAsync is mandatory.

You can trigger the default UI experience with RecognizeWithUIAsync. This opens the UI and starts waiting for you to speak. Eventually it returns the result as a SpeechRecognitionResult that holds the recognized text, together with extra information such as the confidence of the answer (as a level in an enumeration, and as a percentage). Here’s the full use case:

/// <summary>
/// Opens the speech recognizer UI.
/// </summary>
private async void OpenUI_Click(object sender, RoutedEventArgs e)
{
    SpeechRecognizer recognizer = new SpeechRecognizer();

    SpeechRecognitionTopicConstraint topicConstraint
            = new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.Dictation, "Development");

    recognizer.Constraints.Add(topicConstraint);
    await recognizer.CompileConstraintsAsync(); // Required

    // Open the UI.
    var results = await recognizer.RecognizeWithUIAsync();
    if (results.Confidence != SpeechRecognitionConfidence.Rejected)
    {
        this.Result.Text = results.Text;
        // No need to call 'Voice.Say'. The control speaks itself.
    }
    else
    {
        this.Result.Text = "Sorry, I did not get that.";
    }
}

This is how the default UI looks like. It takes the top half of the screen:

 phone_UI

If you don’t like this UI, then you can start a speech recognition session using RecognizeAsync. The recognized text is revealed in the Completed event of the resulting IAsyncOperation. Here’s the whole UI-less story:

/// <summary>
/// Starts a speech recognition session.
/// </summary>
private async void Listen_Click(object sender, RoutedEventArgs e)
{
    this.Result.Text = "Listening...";

    SpeechRecognizer recognizer = new SpeechRecognizer();

    SpeechRecognitionTopicConstraint topicConstraint
            = new SpeechRecognitionTopicConstraint(SpeechRecognitionScenario.Dictation, "Development");

    recognizer.Constraints.Add(topicConstraint);
    await recognizer.CompileConstraintsAsync(); // Required

    var recognition = recognizer.RecognizeAsync();
    recognition.Completed += this.Recognition_Completed;
}

/// <summary>
/// Speech recognition completed.
/// </summary>
private async void Recognition_Completed(IAsyncOperation<SpeechRecognitionResult> asyncInfo, AsyncStatus asyncStatus)
{
    var results = asyncInfo.GetResults();

    if (results.Confidence != SpeechRecognitionConfidence.Rejected)
    {
        await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, new DispatchedHandler(
            () => { this.Result.Text = results.Text; ; }));
     }
    else
    {
        this.Result.Text = "Sorry, I did not get that.";
    }
}

This was just an introduction, there’s a lot more in the Windows.Media.SpeechRecognition namespace.

Windows App

The SpeechRecognizer class is only native to Windows Phone apps. But don’t worry: if you download the Bing Speech Recognition Control for Windows 8.1, you get roughly the same experience. Before you can actually use the control, you need to register an application in the Azure Market Place to get the necessary credentials. Under the hood the control delegates the processing to a web service that relies on OAUTH. Here’s a screenshot of the application registration pages:

RegisterApplication1

RegisterApplication2

If you’re happy with the standard UI then you just drop the control on a page, like this:

<sp:SpeechRecognizerUx x:Name="SpeechControl" />

Before starting a speech recognition session, you have to provide your credentials:

var credentials = new SpeechAuthorizationParameters();
credentials.ClientId = "YourClientIdHere";
credentials.ClientSecret = "YourClientSecretHere";
this.SpeechControl.SpeechRecognizer = new SpeechRecognizer("en-US", credentials);

The rest of the story is similar to the Windows Phone code. Here’s how to open the standard UI:

/// <summary>
/// Activates the speech control.
/// </summary>
private async void OpenUI_Click(object sender, RoutedEventArgs e)
{
    // Always call RecognizeSpeechToTextAsync from inside
    // a try block because it calls a web service.
    try
    {
        var result = await this.SpeechControl.SpeechRecognizer.RecognizeSpeechToTextAsync();
        if (result.TextConfidence != SpeechRecognitionConfidence.Rejected)
        {
            ResultText.Text = result.Text;
            var voice = new Voice();
            voice.Say(result.Text);
        }
        else
        {
            ResultText.Text = "Sorry, I did not get that.";
        }
    }
    catch (Exception ex)
    {
        // Put error handling here.
    }
}

This is how the UI looks like. I put a a red box around it:

BingUI

If you want to skip (or replace) the UI part, then you don’t need the control in your XAML (but mind that you still have to download and install it). Just prepare a SpeechRecognizer instance:

// The custom speech recognizer UI.
SR = new SpeechRecognizer("en-US", credentials);
SR.AudioCaptureStateChanged += SR_AudioCaptureStateChanged;
SR.RecognizerResultReceived += SR_RecognizerResultReceived;

And call RecognizeSpeechToTextAsync when you’re ready:

/// <summary>
/// Starts a speech recognition session through the custom UI.
/// </summary>
private async void ListenButton_Click(object sender, RoutedEventArgs e)
{
    // Always call RecognizeSpeechToTextAsync from inside
    // a try block because it calls a web service.
    try
    {
        // Start speech recognition.
        var result = await SR.RecognizeSpeechToTextAsync();

        // Write the result to the TextBlock.
        if (result.TextConfidence != SpeechRecognitionConfidence.Rejected)
        {
            ResultText.Text = result.Text;
        }
        else
        {
            ResultText.Text = "Sorry, I did not get that.";
        }
    }
    catch (Exception ex)
    {
        // If there's an exception, show the Type and Message.
        ResultText.Text = string.Format("{0}: {1}",
            ex.GetType().ToString(), ex.Message);
    }
}

/// <summary>
/// Cancels the current speech recognition session.
/// </summary>
private void CancelButton_Click(object sender, RoutedEventArgs e)
{
    SR.RequestCancelOperation();
}

/// <summary>
/// Stop listening and start thinking.
/// </summary>
private void StopButton_Click(object sender, RoutedEventArgs e)
{
    SR.StopListeningAndProcessAudio();
}

/// <summary>
/// Update the speech recognition audio capture state.
/// </summary>
private void SR_AudioCaptureStateChanged(SpeechRecognizer sender, SpeechRecognitionAudioCaptureStateChangedEventArgs args)
{
    this.Status.Text = args.State.ToString();
}

/// <summary>
/// A result was received. Whether or not it is intermediary depends on the capture state.
/// </summary>
private void SR_RecognizerResultReceived(SpeechRecognizer sender, SpeechRecognitionResultReceivedEventArgs args)
{
    if (args.Text != null)
    {
        this.ResultText.Text = args.Text;
    }
}

For Windows as well as Phone projects that require speech-to-text, don’t forget to enable the ‘Microphone’ capability. That also means that the user will have to give consent (only once):

Capabilities

Speech Synthesis

You now know how to let your device listen to you, it’s time to give it a voice. For speech synthesis (text-to-speech) both Phone and Windows apps have access to a Universal SpeechSynthesizer class. The SynthesizeTextToStreamAsync method transforms text into an audio stream (like a *.wav file). You can optionally provide the voice to be used; by selecting from the list of voices (SpeechSynthesizer.AllVoices) that are installed on the device. Each of these voices has it own gender and language. The voices depend on the device and your cultural settings: my laptop only speaks US and UK English, but my phone seems to be fluent in French and German too.

When the speech synthesizer's audio stream is complete, you can play it via a MediaElement on the page. Note that Silverlight 8.1 apps do not need a MediaElement, they can call the SpeakTextAsync method, which is not available for Universal apps.

Here’s the full flow. I wrapped it in a Voice class in the shared project (in an MVVM app this would be a 'Service'):

/// <summary>
/// Creates a text stream from a string.
/// </summary>
public void Say(string text, int voice = 0)
{
    var synthesizer = new SpeechSynthesizer();
    var voices = SpeechSynthesizer.AllVoices;
    synthesizer.Voice = voices[voice];

    var spokenStream = synthesizer.SynthesizeTextToStreamAsync(text);

    spokenStream.Completed += this.SpokenStreamCompleted;
}

/// <summary>
/// The spoken stream is ready.
/// </summary>
private async void SpokenStreamCompleted(IAsyncOperation<SpeechSynthesisStream> asyncInfo, AsyncStatus asyncStatus)
{
    var mediaElement = this.MediaElement;

    // Make sure to be on the UI Thread.
    var results = asyncInfo.GetResults();
    await mediaElement.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, new DispatchedHandler(
        () => { mediaElement.SetSource(results, results.ContentType); }));
}

In a multi-page app, you must make sure that there is a MediaElement on each page. In the sample app, I reused a technique from a previous blog post. I created a custom template for the root Frame:

<Application.Resources>
    <!-- Injecting a media player on each page -->
    <Style x:Key="RootFrameStyle"
            TargetType="Frame">
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="Frame">
                    <Grid>
                        <!-- Voice -->
                        <MediaElement IsLooping="False" />
                        <ContentPresenter />
                    </Grid>
                </ControlTemplate>
            </Setter.Value>
        </Setter>
    </Style>
</Application.Resources>

And applied it in app.xaml.cs:

// Create a Frame to act as the navigation context and navigate to the first page
rootFrame = new Frame();

// Injecting media player on each page.
rootFrame.Style = this.Resources["RootFrameStyle"] as Style;

// Place the frame in the current Window
Window.Current.Content = rootFrame;

The voice class can now easily access the MediaElement from the current page:

public Voice()
{
    DependencyObject rootGrid = VisualTreeHelper.GetChild(Window.Current.Content, 0);
    this.mediaElement = (MediaElement)VisualTreeHelper.GetChild(rootGrid, 0) as MediaElement;
}

/// <summary>
/// Gets the MediaElement that was injected into the page. 
/// </summary>
private MediaElement MediaElement
{
    get
    {
        return this.mediaElement;
    }
}

The speech synthesizer can also generate an audio stream from Speech Synthesis Markup Language (SSML). That’s an XML format in which you can not only write the text to be spoken, but also the pauses, the changes in pitch, language, or gender, and how to deal with dates, times, and numbers, and even detailed pronunciation through phonemes. Here’s the document from the sample app:

<?xml version="1.0" encoding="utf-8" ?>
<speak version='1.0'
       xmlns='http://www.w3.org/2001/10/synthesis'
       xml:lang='en-US'>
  Your reservation for <say-as interpret-as="cardinal"> 2 </say-as> rooms on the <say-as interpret-as="ordinal"> 4th </say-as> floor of the hotel on <say-as interpret-as="date" format="mdy"> 3/21/2012 </say-as>, with early arrival at <say-as interpret-as="time" format="hms12"> 12:35pm </say-as> has been confirmed. Please call <say-as interpret-as="telephone" format="1"> (888) 555-1212 </say-as> with any questions.

  <voice gender='male'>
    <prosody pitch='x-high'> This is extra high pitch. </prosody >
    <prosody rate='slow'> This is the slow speaking rate. </prosody>
  </voice>
  <voice gender='female'>
    <s>Today we preview the latest romantic music from Blablabla.</s>
  </voice>

  This is an example of how to speak the word <phoneme alphabet='x-microsoft-ups' ph='S1 W AA T . CH AX . M AX . S2 K AA L . IH T'> whatchamacallit </phoneme>.

  <voice gender='male'>
    For English, press 1.
  </voice>

  <!-- Language switch:  does not work if you do not have a french voice. -->
  <voice gender='female' xml:lang='fr-FR'>
    Pour le français, appuyez sur 2
  </voice>

</speak>

Here’s how to generate the audio stream from it:

/// <summary>
/// Creates a text stream from an SSML string.
/// </summary>
public void SaySSML(string text, int voice = 0)
{
    var synthesizer = new SpeechSynthesizer();
    var voices = SpeechSynthesizer.AllVoices;
    synthesizer.Voice = voices[voice];

    var spokenStream = synthesizer.SynthesizeSsmlToStreamAsync(text);

    spokenStream.Completed += this.SpokenStreamCompleted;
}

Speech Recognition User Control

The Windows Phone project in the sample app contains a SpeechInputBox. It’s a user control for speech input through voice or keyboard that is inspired by the Cortana look-and-feel. It comes with dependency properties for the text and a highlight color, and it raises an event when the input is processed:

SpeechInputBox

The control behaves like the Cortana one:

  • it starts listening when you tap on the microphone icon,
  • it opens the onscreen keyboard if you tap on the text box,
  • it notifies you when it listens to voice input,
  • the arrow button puts the control in thinking mode, and
  • the control says the result out loud when the input came from voice (not from typed input).

The implementation is very straightforward: depending on the state the control shows some UI elements, and calls the API’s that were already discussed in this article. There’s definitely room for improvement: you could add styling, animations, sound effects, and localization.

Here’s how to use the control in a XAML page. I did not use data binding in the sample, the main page hooked an event handler to TextChanged:

<Controls:SpeechInputBox x:Name="SpeechInputBox" Highlight="Cyan" />

Here are some screenshot of the SpeechInputBox in action, next to its Cortana inspiration:

Cortana_InputBox ctl_initial ctl_listening ctl_thinking ctl_typing ctl_result

If you intend to build a Silverlight 8.1 version of this control, you might want to start from the code in the MSDN Voice Search for Windows Phone 8.1 sample (there are quite some differences with Universal apps: in the XAML as well as in the API calls). If you want to roll your own UI, make sure you follow the Speech Design Guidelines.

Code

Here’s the code, it was written in Visual Studio 2013 Update 3. Remember that you need to register an app to get the necessary credentials to run the Bing Speech Recognition control: U2UC.WinUni.SpeechSample.zip (661.3KB).

Enjoy!

XAML Brewer

Universal Windows Apps: a Tale of Two Calendars.

It was the best of times,
it was the worst of times,
it was the age of wisdom,
it was the age of foolishness,
it was the server side calendar on the tablet,
it was the client side calendar on the smart phone.

This article explains two ways of managing appointments in your calendar through a Universal Windows app. I created a small sample MVVM app that hosts a Calendar service with the following functionality against the user’s default calendar:

  • Open the calendar at a specific date and time,
  • create a new appointment,
  • open the calendar at the newly created appointment,
  • delete the newly created appointment,
  • display the number of appointments that were created by the sample app,
  • display the date and time of the appointments that were created by the sample app, and
  • delete all appointments that were created by the sample app.

Here’s a screenshot of it – the orange icons are the buttons that fire the commands:

MainBoth

The Windows Phone part was built upon the Windows 8.1 Appointments API that is elaborated in this article on the Windows App Builder blog: Build apps that connect with People and Calendar, Part 2: Appointments. That API hosts the static AppointmentManager class, which interacts with the user’s Appointments provider app -by default the Calendar app- at the UI level. Unfortunately, most of the AppointmentManager’s methods are only applicable to the phone and cannot be called from a Store app on a tablet or PC. Here are the methods that are called in the phone project of the sample app. The one to show the details of an appointment cannot be used in a Store app, the others are really universal:

  • ShowTimeFrameAsync: displays a time frame from an appointments calendar through the Appointments provider app's primary UI.
  • ShowAddAppointmentAsync: opens the Appointments provider Add Appointment UI, to enable the user to add an appointment.
  • ShowAppointmentDetailsAsync: opens the Appointments provider Appointment Details UI on the specified appointment.
  • ShowRemoveAppointmentAsync: opens the Appointments provider Remove Appointment UI, to enable the user to remove an appointment.

Don’t forget to declare the Appointments capability in the Phone app, or it won’t work. Strangely enough that capability does not exist for Store apps, there you can always use [the supported parts of] this API.

The appointments are not created or removed by the calling app itself, that task is delegated to the Appointments provider. The Appointments provider acts on the local calendar on the device. The local appointment identifiers are passed back to the app, and the appointments themselves are later synced up to the user’s default calendar –most probably Outlook- where they get their ‘real’ identifier (and yes, that’s a different ID).

The API is nice and easy. Here’s the call to open the user’s calendar app at a specific date and time. I embedded it in the Calendar service in the sample app:

public async static Task Open(DateTimeOffset dto, TimeSpan ts)
{
    await AppointmentManager.ShowTimeFrameAsync(dto, ts);
}

Here’s how the ViewModel opens the calendar:

private async void Open_Executed()
{
    await Calendar.Open(DateTimeOffset.Now.AddDays(-7), TimeSpan.FromHours(1));
}

And this is how it looks like on the lightest phone emulator:

OpenPhone

To add or replace an appointment using the Windows 8.1 Appointments API, you need to provide an instance of the Appointment class. Here’s how the ViewModel of the sample app creates one of these:

var appt = new Appointment();
appt.Subject = "Exterminate Enemy";
appt.Details = "Destroy an extraterrestrial race of human-sized pepper shakers, each equipped with a single mechanical eyestalk mounted on a rotating dome, a gun mount containing an energy weapon and a telescopic manipulator arm which is usually tipped by an appendage resembling a sink plunger.";
appt.Location = "That planet on which silence will fall when the oldest question in the universe is asked.";
appt.Invitees.Add(new AppointmentInvitee() { DisplayName = "That impossible girl.", Address = "All around." });
appt.Duration = TimeSpan.FromMinutes(50);
appt.StartTime = DateTimeOffset.Now.AddDays(7);

These are the methods of the sample Calendar service that allow you to add a new appointment. Depending on the device, the Appointments provider will take the whole screen, or appear in a Popup – that’s why you need to provide the Rectangle parameter:

public async static Task<string> Add(Appointment appt)
{
    var selection = new Rect(new Point(Window.Current.Bounds.Width / 2, Window.Current.Bounds.Height / 2), new Size());
    return await Add(appt, selection);
}

public async static Task<string> Add(Appointment appt, Rect selection)
{
    var id = await AppointmentManager.ShowAddAppointmentAsync(appt, selection, Placement.Default);
    AddAppointmentId(id);
    if (String.IsNullOrEmpty(id))
    {
        Toast.ShowInfo("The appointment was not added.");
    }

    return id;
}

These calls return the local identifier of the appointment, which is only valid on the current device. This might be the first show stopper against using this API in your app. Furthermore, the API has no means for querying the calendar – like ‘give me all appointment id’s for appointments that have Time Travel as subject’. So if you require that in your app, please look for another API – there are plenty of them.

I decided to keep the list of identifiers in a semicolon-separated string in the app’s Roaming Settings:

public static void AddAppointmentId(string appointmentId)
{
    if (String.IsNullOrEmpty(appointmentId))
    {
        return;
    }

    string ids = ApplicationData.Current.RoamingSettings.Values["AppointmentIds"] as string;
    if (String.IsNullOrEmpty(ids))
    {
        ids = appointmentId;
    }
    else
    {
        ids += ";" + appointmentId;
    }

    ApplicationData.Current.RoamingSettings.Values["AppointmentIds"] = ids;
}
public static List<string> AppointmentIds
{
    get
    {
        string ids = ApplicationData.Current.RoamingSettings.Values["AppointmentIds"] as string;
        if (String.IsNullOrEmpty(ids))
        {
            return new List<string>();
        }

        return new List<string>(ids.Split(';'));
    }
}

To view or delete an appointment, you need to provide the local identifier. Here’s the code from the Calendar service that opens the Appointment provider on a to-be-deleted appointment. The second parameter enables an On Error Resume Next Whistlingscenario – useful when you want to delete all appointments that were created:

public async static Task Delete(string appointmentId, bool ignoreExceptions = false)
{
    var selection = new Rect(new Point(Window.Current.Bounds.Width / 2, Window.Current.Bounds.Height / 2), new Size());
    try
    {
        var success = await AppointmentManager.ShowRemoveAppointmentAsync(appointmentId, selection);
    }
    catch (Exception)
    {
        if (!ignoreExceptions)
        {
            throw;
        }
    }
}

Here are some screen shots of AppointmentManager calls on the phone emulator – respectively to add, show the details of, and remove an appointment:

PhoneNew SavePhone DeletePhone

Unfortunately the ShowAppointmentDetailsAsync and the ShowEditNewAppointmentAsync calls apply only the phone. You can’t use them in Windows Store apps – although they’re in a Universal API. In some use cases, that’s good enough. If you built a Universal Store app that comes with an accompanying phone app that adds appointment management, then the Windows 8.1 Appointments API is what you need. If you want the exact same functionality on all device types, then you have to look for another API.

The only AppointmentManager call that is really useful in a Windows Store app, is ShowTimeFrameAsync - the call to open the calendar to a specific date and time. This is how the result looks like in the sample app:

StoreOpen

This is how a call to ShowAddAppointmentAsync looks like. Some information of the to-be-inserted appointment appears in a Popup that carries the color scheme of the Calendar app. The only thing you can control is the position of the Popup:

StoreApptMgrUI

So some parts of the Windows 8.1 Appointments API don’t work on a tablet or PC, while other parts simply look ugly on these platforms. Now, there are a lot of alternative appointment API’s available: you can use the Universal AppointmentStore and AppointmentCalendar classes, or the non-universal citizens of the Microsoft.Phone.UserData namespace – including a very promising Appointments class that comes with query capabilities. Unfortunately these API’s have one thing in common: they only apply to the phone.

For the Windows Store app part of the sample app, I decided to directly appeal to the root of the user’s calendar –Outlook.com- and use the Live Connect API for this. This starts with installing the Live SDK, and associating the app with the Store. Just read the first paragraphs of my article on OneDrive integration for more details on this procedure. Since you’re directly talking to the calendar(s) on the server(s), you need to be logged in:

private static async Task LogIn()
{
    if (NetworkInformation.GetInternetConnectionProfile().GetNetworkConnectivityLevel() >= NetworkConnectivityLevel.InternetAccess)
    {
        LiveAuthClient auth = new LiveAuthClient();
        var loginResult = await auth.LoginAsync(new string[] { "wl.calendars", "wl.calendars_update", "wl.events_create" });
        if (loginResult.Session != null)
        {
            client = new LiveConnectClient(loginResult.Session);
        }

        isLoggedIn = (loginResult.Status == LiveConnectSessionStatus.Connected);
    }
    else
    {
        isLoggedIn = false;
    }
}

You also have to know the identifier of the user’s default calendar. Here’s how to find it:

private async static Task<string> FetchCalendarId()
{
    if (!isLoggedIn)
    {
        await LogIn();
    }

    if (client == null || !isLoggedIn)
    {
        return string.Empty;
    }

    try
    {
        LiveOperationResult lor = await client.GetAsync("me/calendars");
        dynamic result = lor.Result;
        foreach (dynamic calendar in result.data)
        {
            // We assume that the first one is the default.
            string id = calendar.id;
            return id;
        }
    }
    catch (Exception ex)
    {
        Debugger.Break();
        return string.Empty;
    }

    return string.Empty;
}

The HTTP POST call to add an appointment takes a dictionary as parameter. So here’s an extension method to format an Appointment instance:

public static Dictionary<string, object> AsDictionary(this Appointment appt)
{
    var calendarEvent = new Dictionary<string, object>();
    calendarEvent.Add("name", appt.Subject);
    calendarEvent.Add("description", appt.Details);
    calendarEvent.Add("start_time", appt.StartTime.ToString("u"));
    calendarEvent.Add("end_time", appt.StartTime.Add(appt.Duration).ToString("u"));
    calendarEvent.Add("location", appt.Location);
    calendarEvent.Add("is_all_day_event", appt.AllDay);

    return calendarEvent;
}

Here’s the full call to add an appointment to the user’s default calendar. It’s not exactly a one-liner: we

  • fetch the calendar id,
  • ensure we’re logged on,
  • add the appointment,
  • store the appointment id in the Roaming Settings,
  • open the local calendar app at the appropriate day, and finally
  • remind the user that he might need to sync it - since we added the appointment on the server-side:
public async static Task<string> Add(Appointment appt)
{
    var calendarId = await FetchCalendarId();

    if (!isLoggedIn)
    {
        await LogIn();
    }

    if (client == null || !isLoggedIn)
    {
        Toast.ShowInfo("Sorry, I could not open your calendar.");
        return string.Empty;
    }

    try
    {
        string parm = string.Format("{0}/events", calendarId);
        LiveOperationResult lor = await client.PostAsync(parm, appt.AsDictionary());
        dynamic result = lor.Result;
        string id = result.id;
        AddAppointmentId(id);
        await Open(appt.StartTime.Date, TimeSpan.FromDays(1));
        Toast.ShowInfo("You may need to sync the calendar.");
        return id;
    }
    catch (Exception ex)
    {
        Debugger.Break();
        return string.Empty;
    }
}

This is how it looks like in the sample app:

StoreToast

When the user taps on the appointment, its details are displayed, with the possibility to edit and delete:

StoreSelect

The Live Connect API comes with an HTTP GET to fetch the appointment details if you know its identifier. Here’s how the sample app fetches the date and time of an appointment:

public async static Task<DateTime> FetchAppointmentDate(string eventId)
{
    if (!isLoggedIn)
    {
        await LogIn();
    }

    if (client == null || !isLoggedIn)
    {
        return DateTime.MinValue;
    }

    try
    {
        LiveOperationResult lor = await client.GetAsync(eventId);
        dynamic result = lor.Result;
        string dt = result.start_time;

        return DateTime.Parse(dt);
    }
    catch (Exception ex)
    {
        // Debugger.Break();
        return DateTime.MinValue;
    }
}

For showing the details of an appointment, I’m only interested in getting its date and time. I use this to open the calendar at the appropriate date:

public async static Task Display(string appointmentId)
{
    var date = await FetchAppointmentDate(appointmentId);
    if (date != DateTime.MinValue)
    {
        await Open(date.Date, TimeSpan.FromDays(1));
    }
    else
    {
        // Not Found.
        Toast.ShowError("Sorry, I could not find the appointment.");
    }
}

Here’s the result in the sample app, this UI should not come as a surprise:

StoreNewAndDisplay

For deleting an appointment, you have two options: open the calendar to the appointment’s date and let the user delete it, or delete it directly with a HTTP DELETE call. Here’s the code for the latter:

public async static Task Delete(string appointmentId, bool ignoreFailure = false)
{
    try
    {
        LiveOperationResult lor = await client.DeleteAsync(appointmentId);
        RemoveAppointmentId(appointmentId);
    }
    catch (Exception ex)
    {
        if (ignoreFailure)
        {
            // On Error Resume Next ...
            RemoveAppointmentId(appointmentId);
        }
        else
        {
            Toast.ShowError("Sorry, something went wrong.");
            Debugger.Break();
        }
    }
}

For the sake of completeness, here’s the shared code that generates the welcome message for Store and Phone apps in the ViewModel. We cannot query for the list of appointments with the used API’s, but we can count the number of identifiers in the Roaming Settings:

        public async Task UpdateMessage()
        {
            message = string.Format("You have {0} appointment(s).", Calendar.AppointmentIds.Count);

#if WINDOWS_PHONE_APP
            this.OnPropertyChanged("Message");
            return;
#else
            foreach (var apptId in Calendar.AppointmentIds)
            {
                var dt = await Calendar.FetchAppointmentDate(apptId);
                if (dt == DateTime.MinValue)
                {
                    message += "\n - (not found)";
                }
                else
                {
                    message += "\n -" + dt.ToString();
                }
            }

            this.OnPropertyChanged("Message");
#endif
        }

Here’s the solution’s structure. The Calendar service is spread over all projects as a partial class (with a large portion in the Shared part), while the XAML of the phone box UI is shared as a user control:

StructureAndManifest

Depending on your use case, you may choose one or the other API for managing appointments in your universal app. But don’t mix them as I did in the sample app. The Windows 8.1 Appointments API works with device-specific identifiers against a local calendar, while the Live Connect API works with global identifiers. The appointments end up in the same place, but there seems to be no way to match the identities.

Here’s the full code of the sample app, it was written with Visual Studio 2013 Update 2. Remember to associate the Windows app with the Store to test the Live Connect API: U2UC.WinUni.Appointments.zip (139.4KB)

Enjoy!

XAML Brewer

Playing sounds in a Universal Windows MVVM app

This article describes how to play foreground and background sounds in a XAML-based Universal Windows app. I know that this sounds easy (pun intended). But I will do this while sticking to the MVVM pattern, which makes it a bit more challenging. I’m not targeting a specific framework here. The described architecture can be applied in MVVM Light, Universal Prism, Caliburn.Micro as well as in your home brewed MVVM library.

Here’s how the attached sample app looks like. It comes with two buttons that trigger a different sound effect, a switch to mute the background sound, and a button to navigate to a new page (very uncommon in a single-page app Happy):SoundsApp

What's the challenge for an MVVM solution to play sounds? Well, a universal XAML app can only make noise through a MediaElement, and that MediaElement should be part of the runtime structure of XAML objects known as the visual tree. As a result of that, only a small number of app components -the currently active Views- have access to it. In most cases however it’s the business logic in the Model or the ViewModel that knows which specific sound effect should be played at which particular moment. So the ‘Sound’ functionality should not be restricted to a some Views, but it should be generally available to all application components and component types.

In most MVVM ecosystems, this type of global functionality ends up in a so-called Service (other examples include logging, authorization, and toast notification). And so does the SoundPlayer: it’s a global service that comes with a Play method. That method has two parameters:

  • a reference to the sound effect (from a developer-friendly enumeration), and
  • a boolean indicating whether the sound should be played in the foreground (once) or in the background (in a loop).

Since a MediaElement can only play one sound at a time, the SoundPlayer is connected to two of them – one for the foreground and one for background. The background player can be disabled (muted) by the app.

Here’s an class diagram of the SoundPlayer, together with the Visual Studio solution structure. The platform specific projects for Windows 8.1 and Windows Phone 8.1 are collapsed, since they’re empty – except for the tile images. All the code is in the Shared project:

Solution

This is how everything was set up. The first challenge is to make sure that every Page of the app is decorated with two MediaElement instances. An easy way to do this, is to put these elements in the app’s root frame. This can be done through a custom style, e.g. in the App.xaml file:

<Application.Resources>
    <!-- Injecting media players on each page --> 
    <Style x:Key="RootFrameStyle"
           TargetType="Frame">
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="Frame">
                    <Grid>
                        <!-- Foreground Player -->
                        <MediaElement IsLooping="False" />
                        <!-- Background Player -->
                        <MediaElement IsLooping="True" />
                        <ContentPresenter />
                    </Grid>
                </ControlTemplate>
            </Setter.Value>
        </Setter>
    </Style>
</Application.Resources>

Make sure that the style is applied when the app is launched, by adding an extra line of code in the standard OnLaunched method in App.xaml.cs:.

protected override void OnLaunched(LaunchActivatedEventArgs e)
{
    Frame rootFrame = Window.Current.Content as Frame;

    if (rootFrame == null)
    {
        rootFrame = new Frame();

        // Injecting media players on each page.
        rootFrame.Style = this.Resources["RootFrameStyle"] as Style;

        // ...
    }

    // ...
}

As long as you navigate within the frame, the media elements remain available. But when you programmatically switch to another root Frame, you have to make sure to apply this style to it. That’s what the navigate button in the sample app does:

var rootFrame = new Frame();
rootFrame.Style = App.Current.Resources["RootFrameStyle"] as Style;
Window.Current.Content = rootFrame ;
rootFrame.Navigate(typeof(MainPage));

Achievement unlocked: every page will always two MediaElement instances on it. We just have to make them available to the SoundPlayer service. This assignment is done in the Initialize call in the SoundPlayer class itself. Make sure to preserve the state since we might be hosted in a new frame with brand new UI elements at their default -unmuted- state:

public void Initialize()
{
    // Register media elements to the Sound Service.
    try
    {
        DependencyObject rootGrid = VisualTreeHelper.GetChild(Window.Current.Content, 0);
        var foregroundPlayer = (MediaElement)VisualTreeHelper.GetChild(rootGrid, 0) as MediaElement;
        var backgroundPlayer = (MediaElement)VisualTreeHelper.GetChild(rootGrid, 1) as MediaElement;

        SoundPlayer.ForegroundPlayer = foregroundPlayer;

        // Keep the state.
        var isMuted = this.IsBackgroundMuted;
        SoundPlayer.BackgroundPlayer = backgroundPlayer;
        this.IsBackgroundMuted = isMuted;
    }
    catch (Exception)
    {
        // Most probably you forgot to apply the custom root frame style.
        SoundPlayer.ForegroundPlayer = null;
        SoundPlayer.BackgroundPlayer = null;
    }
}

Every Page should make a call to this Initialize after loading, so I factored out the call into a common base class for all of the app’s Page-type views.

public class ViewBase : Page
{
    public ViewBase()
    {
        this.Loaded += this.OnLoaded;
    }

    protected virtual void OnLoaded(object sender, RoutedEventArgs e)
    {
        SoundPlayer.Instance.Initialize();
    }
}
public sealed partial class MainPage : ViewBase
{
    // ...
}

To make the SoundPlayer service globally accessible, it was implemented as a Singleton. A static class would not work, since property-changed notification requires an instance. In most MVVM frameworks this instance would be served to you by Dependency Injection, or it would be listening to a Messenger of some sort. Here’s the core class definition:

internal class SoundPlayer : BindableBase
{
    private static SoundPlayer instance = new SoundPlayer();
    private static MediaElement ForegroundPlayer { get; set; }
    private static MediaElement BackgroundPlayer { get; set; }

    public static SoundPlayer Instance
    {
        get
        {
            return instance;
        }
    }

    public bool IsBackgroundMuted
    {
        get
        {
            if (BackgroundPlayer == null)
            {
                return false;
            }

            return BackgroundPlayer.IsMuted;
        }

        set
        {
            if (BackgroundPlayer != null)
            {
                BackgroundPlayer.IsMuted = value;
                this.OnPropertyChanged("IsBackgroundMuted");
            }
        }
    }

    public async Task Play(Sounds sound, bool inBackground = false)
    {
        var mediaElement = inBackground ? BackgroundPlayer : ForegroundPlayer;

        if (mediaElement == null)
        {
            return;
        }

        string source = string.Format("ms-appx:///Assets/{0}.mp3", sound.ToString());
        mediaElement.Source = new Uri(source);

        await mediaElement.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
        {
            mediaElement.Stop();
            mediaElement.Play();
        });
    }
}

The Play method was made asynchronous to keep the UI responsive. The list of available sound effects is contained in an enumeration that maps the physical mp3 assets that you can see in the previous screen shot. That makes it easy for all components to select the sound effect they want to play:

public enum Sounds
{
    Nature,
    Sweep,
    Bell
}

By the way, if you’re looking for royalty-free sound effects, you may want to check SoundGator or SoundBible.

The SoundPlayer is now accessible from the View components. So you could decide to start some background music after a page is loaded:

protected async override void OnLoaded(object sender, RoutedEventArgs e)
{
    base.OnLoaded(sender, e);
    await SoundPlayer.Instance.Play(Sounds.Nature, true);
}

The SoundPlayer is also accessible from the (View)Model components. Here's the code for the Bell command in the MainViewModel:

private async void Bell_Executed()
{
    await SoundPlayer.Instance.Play(Sounds.Bell);
}

None of these app components are actually aware of the MediaElement instances that are doing the work. That knowledge is encapsulated in the custom Frame style and the SoundPlayer itself.

Here’s the full source code, it was written with Visual Studio 2013 Update 2. Feel free to adapt it to your favorite MVVM framework: U2UC.WinUni.Sound.zip (1.3MB)

Enjoy!

XAML Brewer

Universal App with Lex.DB

This article shows how to create a Universal Windows App that stores its local data in Lex.DB. This is a lightweight, developer-friendly in-process database engine, completely written in C#. For an introduction to building Store Apps on top of Lex.DB, please check this article of mine. For a more advanced dive into performance tuning, check this one – and make sure you don’t skip the valuable comments from Lex Lavnikov, the author of Lex.DB, at the end.

Lex.DB can be used on .NET 4+, Silverlight 5+, Windows Phone 8+, WinRT 8+, and Xamarin. Recently this alternative for SQLite was upgraded to support Universal Windows Apps.

I created a straightforward sample app, based on the Universal App with SQLite blog post by Nicolò Carandini. I added a tiny MVVM Framework with BindableBase and RelayCommand just for fun. The sample app manages a list of Person instances.

This is the Person class, as simple as can be:

public class Person
{
    public int Id { get; set; }

    public string Name { get; set; }

    public string Degree { get; set; }

    public string Occupation { get; set; }
}

The sample app comes with commands to add a Person, delete the selected Person, and reset the database to its default content. Here’s how it looks like in the emulator and the simulator:

EmulatorAndSimulator

You just need to add the Lex.DB Nuget package in your solution:

LexdbNuget

Each of the platform-specific projects will reference its own Lex.DB dll. That’s a lot simpler than SQLite, where you need to install an SDK, reference the C++ runtime *and* integrate some extra source code into your projects. The following screenshots illustrate the impact of both databases to your Visual Studio solution, with SQLite on the left, and Lex.DB on the right:

ProjectReferences

Here’s how the data access layer creates a reference to the database – in a static constructor:

private static DbInstance db;

static Dal()
{
    // Create database
    db = new DbInstance("Storage", ApplicationData.Current.RoamingFolder);

    // Define table mapping
    // * 1st parameter is primary key
    // * 2nd parameter is autoGen e.g. auto-increment
    db.Map<Person>().Automap(p => p.Id, true);

    // Initialize database
    db.Initialize();
}

The following method returns the content of the Person table:

public static IEnumerable<Person> GetPeople()
{
    return db.Table<Person>();
}

I defined two methods to insert/update Person instances: one for a single instance (it returns the generated primary key) and another one to save a list (in one transaction – read the already mentioned perfomance tuning article for more details):

public static int SavePerson(Person person)
{
    db.Table<Person>().Save(person);
    return person.Id;
}

public static Task SavePeople(IEnumerable<Person> people)
{
    return db.Table<Person>().SaveAsync(people);
}

Here’s how to give your database some initial (default) content:

public static Task ResetPeople()
{
    // Clear
    db.Purge<Person>();

    // Repopulate
    return Dal.SavePeople(
        new List<Person>()
        {
            new Person() { Name="Leonard Leakey Hofstadter", Degree="Ph.D.", Occupation="Experimental physicist"},
            new Person() {Name="Sheldon Lee Cooper", Degree="Ph.D.", Occupation="Theoretical physicist"},
            new Person() {Name="Howard Joel Wolowitz", Degree="M.Eng.", Occupation="Aerospace engineer"},
            new Person() {Name="Rajesh Ramayan Koothrappali", Degree="Ph.D.", Occupation="Astrophysicist"}
        });
}

For the sake of completeness, here’s the delete method:

public static Task DeletePeople(IEnumerable<Person> people)
{
    return db.Table<Person>().DeleteAsync(people);
}

Here’s the full source code of the sample app, it was written in Visual Studio 2013 Update 2: U2UC.WinUni.LexDBSample.zip (1.2MB)

Enjoy!

XAML Brewer

A Marching Ants Animation for Universal Windows Apps

ants-line In a lot of apps we need to draw lines on some kind of map to display a route. If you want such line to also indicate the driving direction and speed, then you could apply the Marching Ants Effect, where you represent the route as a dotted or dashed line and let the dashes walk slowly sideways and up and down. In the XAML world, this is remarkably easy. All you need to do is apply a dash pattern to the line (or PolyLine, or any other Shape) through the Shape.StrokeDashArray property and then animate its Shape.StrokeDashOffset. Here’s an example of the effect – since a screenshot would be rather silly, I created a movie where you see the marching ants (well, in this case they might be orcs) in the attached sample project: MarchingAnts.wmv (1.4MB)

As mentioned, you have to first make the line look as an ants line, so use the appropriate values for the Shape.StrokeDashCap and Shape.StrokeLineJoin properties:

Polyline line = new Polyline();

// Add Points
// line.Points.Add(new Point(...));

line.Stroke = new SolidColorBrush(Colors.OrangeRed);
line.StrokeThickness = 18;
line.StrokeDashArray = new DoubleCollection() { 4, 2 };
line.StrokeDashCap = PenLineCap.Round;
line.StrokeLineJoin = PenLineJoin.Round;

For the animation I use a Storyboard with nothing but a DoubleAnimation on the Shape.StrokeDashOffset property. That offset moves from 0 to the total length of the dash pattern, which can be conveniently calculated with a LINQ Sum operator. I implemented it as an extension method to the Shape class. It only takes the duration of the animation as a parameter:

public static void ApplyMarchingAntsAnimation(this Shape shape, TimeSpan duration)
{
    Storyboard storyboard = new Storyboard();
    DoubleAnimation doubleAnimation = new DoubleAnimation();
    doubleAnimation.From = 0.0;
    doubleAnimation.To = -shape.StrokeDashArray.Sum();
    doubleAnimation.Duration = new Duration(duration);
    doubleAnimation.AutoReverse = false;
    doubleAnimation.RepeatBehavior = RepeatBehavior.Forever;
    doubleAnimation.EnableDependentAnimation = true; // Don't forget
    storyboard.Children.Add(doubleAnimation);
    Storyboard.SetTarget(doubleAnimation, shape);
    Storyboard.SetTargetProperty(doubleAnimation, "StrokeDashOffset");
    storyboard.Begin();
}

You can apply the animation to the Stroke of any Shape with the following one-liner:

line.ApplyMarchingAntsAnimation(TimeSpan.FromSeconds(1));

Everything is implemented in the shared part of a universal app, so it works on the desktop, the tablet and the phone too. This is just a screenshot, but I assure you “it’s alive…”:

MarchingAntsScreens

Here’s the whole source, it was written in Visual Studio 2013 Update 2: U2UC.WinUni.MarchingAnts.zip (555.9KB)

Enjoy!

XAML Brewer

A Show-All-Or-Nothing Behavior for Windows Universal Apps

Universal apps need to have a responsive UI that adapts to hugely differing form factors. This article describes how to build a Border control that hides its content when it’s too large to be fully displayed on the screen. It is implemented as a behavior, since –unlike in WPF- all native WinRT and Windows Phone controls are sealed. Originally, I had built a HideOnTrimTextBlock: a TextBlock that displays nothing if the text becomes too wide to display. The calculation was triggered when the TextBlock’s size changed. I soon discovered two things:

  • the SizeChanged event for a TextBlock is often swallowed by its parent, so you have to walk up the visual tree to hook event handlers there, and
  • the code used only members of FrameworkElement, so it was applicable to more than just TextBlock controls.

So I turned the HideOnTrimTextBlock into a ShowAllOrNothingBorder.

To detect whether the content of an element is too wide to be displayed, we compare its DesiredSize (don’t forget to Measure first) with its rendered ActualWidth.

I implemented this test as an extension method:

public static class FrameworkElementExtensions
{
    /// <summary>
    /// Returns whether or not the content of the element is too wide to be displayed entirely.
    /// </summary>
    public static bool IsContentTooWide(this FrameworkElement element)
    {
        element.Measure(new Size(double.MaxValue, double.MaxValue));
        return element.DesiredSize.Width > (element.ActualWidth + 1);
    }
}

I only check the Width here, feel free to bring the Height into the equation.

The ShowAllOrNothing Universal Behavior implements IBehavior and applies to any Border. In the SizeChanged we verify whether the content is too wide or not, and adjust the Opacity of the border's Child (alternatively you could play on the Visibility). Here’s the whole behavior:

/// <summary>
/// A behavior that makes a Border's content disappear when it doesn't entirely fit the screen.
/// </summary>
public class ShowAllOrNothingBehavior : DependencyObject, IBehavior
{
    private Border border;

    public DependencyObject AssociatedObject
    {
        get { return this.border; }
    }

    public void Attach(DependencyObject associatedObject)
    {
        if (associatedObject is Border)
        {
            this.border = associatedObject as Border;
            this.border.SizeChanged += this.Border_SizeChanged;
        }
        else
        {
            throw new Exception("ShowAllOrNothingBehavior Behavior only applies to Border.");
        };
    }

    private void Border_SizeChanged(object sender, SizeChangedEventArgs e)
    {
        if (this.border.IsContentTooWide())
        {
            this.border.Child.Opacity = 0;
        }
        else
        {
            this.border.Child.Opacity = 1;
        }
    }

    public void Detach()
    {
        if (this.border != null)
        {
            this.border.SizeChanged -= this.Border_SizeChanged;
        }
    }
}

Here’s how to attach the behavior in XAML:

<Page ...
      xmlns:interactivity="using:Microsoft.Xaml.Interactivity"
      xmlns:behaviors="using:U2UC.WinUni.Behaviors"
      ...>

<Border>
    <interactivity:Interaction.Behaviors>
        <behaviors:ShowAllOrNothingBehavior />
    </interactivity:Interaction.Behaviors>
    <TextBlock Text="I'm sorry, Dave. I'm afraid I can't do that." />
</Border>

Here’s a screen shot of the attached sample app. It illustrates a number of text block responsiveness options, such as word wrapping, character trimming, word trimming, and shrinking. The ShowAllOrNothing behavior is attached to the last one:

app_1

The slider at the bottom of the screen determines the width of the text blocks’ parent. When you slide it to the left you’ll see the last text immediately disappear when it’s touched by the red line:

app_2

In some use cases it makes sense to entirely hide a control if it doesn’t fit the screen. In the following screen shot, I believe that none of the text blocks actually produce any useful output:

app_3

Since it’s a Universal app, it’s also supposed to work on the phone:

phone_1 phone_2

All the code is sitting in the Shared project of a Universal App solution. The Windows 8.1 and Windows Phone 8.1 apps just need to reference their own platform-specific Behaviors SDK:

UniversalBehaviorSolution

You may want to implement this behavior in a Portable Class Library instead. In that case I suggest you first read this article by Joost van Schaik.

For the sake of completeness, here’s how this code would look like in WPF. Since the control classes are not sealed, we can put everything in a Border subclass:

// Wpf Control.
namespace WpfApplication
{
    using System.Windows;
    using System.Windows.Controls;

    /// <summary>
    /// A Border that makes its content disappear when it doesn't entirely fit the screen.
    /// </summary>
    public class ShowAllOrNothingBorder : Border
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="ShowAllOrNothingBorder"/> class.
        /// </summary>
        public ShowAllOrNothingBorder()
        {
            this.SizeChanged += ShowAllOrNothingBorder_SizeChanged;
        }

        /// <summary>
        /// Determines whether content is too wide.
        /// </summary>
        public bool IsContentTooWide()
        {
            this.Measure(new Size(double.MaxValue, double.MaxValue));
            return this.DesiredSize.Width > (this.ActualWidth + 1);
        }

        private void ShowAllOrNothingBorder_SizeChanged(object sender, SizeChangedEventArgs e)
        {
            if (IsContentTooWide())
            {
                this.Child.Opacity = 0;
            }
            else
            {
                this.Child.Opacity = 1;
            }
        }
    }
}

Here’s the full source code, it was written in Visual Studio 2013 Update 2: U2UC.WinUni.Behavior.zip (132.2KB)

Enjoy!

XAML Brewer

Tracking with Tiles and Toasts

This article shows a way to implement multiple alarm clock functionality in a Windows (Store or Enterprise) MVVM XAML App. We’ll use locally scheduled Toast and Tile Notifications to track the progress of a workflow - like a workout scheme or a cooking recipe. The workflow can be started, paused, restarted, and canceled by the user. The app will schedule, reschedule, and unschedule the appropriate Toast and Tile Notifications so that the process can be monitored by the end user on the machine, even when the app gets suspended. Here’s a screenshot of the sample app, it monitors a classic cooking recipe:

Start

When the app is started for the first time, it creates the workflow steps, and welcomes you with a regular toast. I use a helper class that can spawn three different toasts. These toasts are a replacement for the classic Info, Warning, and Error message boxes. Here’s the whole class, I already use it in several published Store apps:

    /// <summary>
    /// Issues Toast Notifications.
    /// </summary>
    public static class Toast
    {
        /// <summary>
        /// Shows the specified text in a toast.
        /// </summary>
        public static void Show(string title, string text)
        {
            Toast.Show(title, text, null);
        }

        /// <summary>
        /// Shows a toast with an info icon.
        /// </summary>
        public static void ShowInfo(string title, string text)
        {
            Toast.Show(title, text, "ms-appx:///Assets/Toasts/Wink.png");
        }

        /// <summary>
        /// Shows a toast with a warning icon.
        /// </summary>
        public static void ShowWarning(string title, string text)
        {
            Toast.Show(title, text, "ms-appx:///Assets/Toasts/Worried.png");
        }

        /// <summary>
        /// Shows a toast with an error icon.
        /// </summary>
        public static void ShowError(string title, string content)
        {
            Toast.Show(title, content, "ms-appx:///Assets/Toasts/Confused.png");
        }

        /// <summary>
        /// Shows a toast with the specified text and icon.
        /// </summary>
        private static void Show(string title, string content, string imagePath)
        {
            XmlDocument toastXml = GetToast(title, content, imagePath);

            ToastNotification toast = new ToastNotification(toastXml);
            ToastNotificationManager.CreateToastNotifier().Show(toast);
        }

        /// <summary>
        /// Gets the toast.
        /// </summary>
        private static XmlDocument GetToast(string title, string content, string imagePath)
        {
            string toastXmlString =
                "<toast>\n" +
                    "<visual>\n" +
                        "<binding template=\"ToastImageAndText02\">\n" +
                            "<image id=\"1\" src=\"" + imagePath + "\"/>\n" +
                            "<text id=\"1\">" + title + "</text>\n" +
                            "<text id=\"2\">" + content + "</text>\n" +
                        "</binding>\n" +
                    "</visual>\n" +
                "</toast>\n";

            XmlDocument toastXml = new XmlDocument();
            toastXml.LoadXml(toastXmlString);

            return toastXml;
        }
    }

Here’s the welcome toast call:

Toast.ShowInfo("Welcome", "I created some default alarms for you.");

And this is how the result looks like:

ToastAtStart

Toast Notifications can be displayed immediately, but they can also be scheduled for the future. Let’s take a look at that, and dive into some other helper classes.

Each step in the chicken recipe workflow is represented by an instance of the Alarm class – the ViewModel to a Toast Notification. An alarm is in one of these states:

    /// <summary>
    /// Alarm States.
    /// </summary>
    public enum AlarmStates
    {
        New,
        Scheduled,
        Paused,
        Canceled,
        Delivered
    }

The Alarm helper class comes with expected properties such as an Identifier, Title, Content, State, and TimeSpan. The TimeLeft is a calculated property:

        /// <summary>
        /// Gets the time left.
        /// </summary>
        public TimeSpan TimeLeft
        {
            get
            {
                switch (this.State)
                {
                    case AlarmStates.New:
                        return this.timeSpan;
                    case AlarmStates.Scheduled:
                        return this.timeSpan.Add(this.enabledAt - DateTime.Now);
                    case AlarmStates.Paused:
                        return this.timeSpan;
                    case AlarmStates.Canceled:
                        return TimeSpan.FromSeconds(0);
                    case AlarmStates.Delivered:
                        return TimeSpan.FromSeconds(0);
                    default:
                        return TimeSpan.FromSeconds(0);
                }
            }
        }

Each Alarm is scheduled when the workflow starts, and has to notify the user when its associated step starts. Here’s the code that schedules an alarm; we update its state and schedule a Toast Notification:

        /// <summary>
        /// Schedules this instance.
        /// </summary>
        public async Task Schedule()
        {
            if (this.state == AlarmStates.Scheduled || this.state == AlarmStates.Delivered)
            {
                // No action.
                return;
            }

            if (this.state == AlarmStates.Paused)
            {
                this.TimeSpan = this.TimeLeft;
            }

            this.enabledAt = DateTime.Now;
            this.ScheduleToast();
            this.State = AlarmStates.Scheduled;

            await RemovePersistedAlarm(this);
        }

The Toast Notification is scheduled through a ToastNotifier that registers a ScheduledToastNotification by calling AddToSchedule. For a introduction to all of these, check the alarm toast notifications sample on MSDN, which also demonstrates Snoozing and Dismiss functionality. Here’s the scheduling source code:

        /// <summary>
        /// The toast notifier
        /// </summary>
        private static ToastNotifier toastNotifier = ToastNotificationManager.CreateToastNotifier();

        /// <summary>
        /// Schedules the toast.
        /// </summary>
        private void ScheduleToast()
        {
            if (this.TimeSpan <= TimeSpan.FromSeconds(0))
            {
                return;
            }

            XmlDocument toastXml = GetToast();
            var toast = new ScheduledToastNotification(toastXml, DateTime.Now.Add(this.TimeSpan));
            toast.Id = this.Id;
            toastNotifier.AddToSchedule(toast);
        }

        /// <summary>
        /// Gets the toast.
        /// </summary>
        private XmlDocument GetToast()
        {
            string toastXmlString =
                "<toast duration=\"long\">\n" +
                    "<visual>\n" +
                        "<binding template=\"ToastImageAndText02\">\n" +
                            "<image id=\"1\" src=\"ms-appx:///Assets/Toasts/AlarmClock.png\"/>\n" +
                            "<text id=\"1\">" + this.title + "</text>\n" +
                            "<text id=\"2\">" + this.content + "</text>\n" +
                        "</binding>\n" +
                    "</visual>\n" +
                    "<audio src=\"ms-winsoundevent:Notification.Looping.Alarm2\" loop=\"true\" />\n" +
                "</toast>\n";

            XmlDocument toastXml = new XmlDocument();
            toastXml.LoadXml(toastXmlString);

            return toastXml;
        }

If you don’t like working with the raw XML version of the toast content, then I suggest you take a look at the NotificationsExtensions project. That project contains a more developer friendly wrapper around all of this.

A scheduled toast notification appears (and sounds) right on time, whether the app is active or not:

BasketToast

ToastNoApp

By the way: scheduled toast notifications –as well as scheduled tile notifications, see further- nicely survive restarts of your machine.

Back to the sample app. The entire workflow as such is not remembered. When the user stops and restarts the app, we recreate the list of alarms based on the scheduled toasts – that’s what GetScheduledToastNotifications does:

        /// <summary>
        /// Returns the scheduled alarms.
        /// </summary>
        public static IEnumerable<Alarm> ScheduledAlarms()
        {
            var toasts = toastNotifier.GetScheduledToastNotifications();

            return from t in toasts
                   select (Alarm)t;
        }

All we have to do is convert each native ScheduledToastNotification back to an instance of our Alarm ViewModel class. I used a conversion operator for that. If you’re dealing with the raw XML representation for the toasts, then you need to apply some XPATH magic (in SelectSingleNode) to get some of the data back:

        /// <summary>
        /// Performs an implicit conversion from <see cref="ScheduledToastNotification"/> to <see cref="Alarm"/>.
        /// </summary>
        public static implicit operator Alarm(ScheduledToastNotification toast)
        {
            Alarm result = new Alarm();

            result.Id = toast.Id;
            result.TimeSpan = toast.DeliveryTime - DateTime.Now;
            result.enabledAt = DateTime.Now;
            result.State = AlarmStates.Scheduled;

            var node = toast.Content.SelectSingleNode("//text[@id=1]");
            if (node != null)
            {
                result.Title = node.InnerText;
            }

            node = toast.Content.SelectSingleNode("//text[@id=2]");
            if (node != null)
            {
                result.Content = node.InnerText;
            }

            return result;
        }

The sample app allows an alarm to be paused, which is something that a native scheduled toast notification can’t handle. When an alarm is paused, we upgrade its state and remove the corresponding toast notification from the schedule:

        /// <summary>
        /// Pauses this instance.
        /// </summary>
        public async Task Pause()
        {
            if (this.state != AlarmStates.Scheduled)
            {
                // No action.
                return;
            }

            this.TimeSpan = this.TimeLeft;
            this.State = AlarmStates.Paused;
            this.UnscheduleToast();

            await AddPersistedAlarm(this);
        }

For unscheduling the toast, we look it up by its identifier through a LINQ query against GetScheduledToastNotifications, and remove it from the list with RemoveFromSchedule:

        /// <summary>
        /// Unschedules the toast.
        /// </summary>
        private void UnscheduleToast()
        {
            var toasts = toastNotifier.GetScheduledToastNotifications();
            var found = (from t in toasts
                         where t.Id == this.Id
                         select t).FirstOrDefault();
            if (found != null)
            {
                toastNotifier.RemoveFromSchedule(found);
            }
        }

For a paused alarm there’s no corresponding ScheduledToastNotification anymore, so when the app is suspended, it would be lost. To prevent this, we serialize the paused alarms as a list in the local folder so we can deserialize it when the app restarts. Here’s the corresponding code for all of this:

        /// <summary>
        /// The serializer
        /// </summary>
        private static AbstractSerializationBase<List<Alarm>> serializer =
            new XmlSerialization<List<Alarm>>()
            {
                FileName = "Alarms.xml",
                Folder = ApplicationData.Current.LocalFolder
            };

        /// <summary>
        /// Returns the persisted alarms.
        /// </summary>
        public static async Task<List<Alarm>> PersistedAlarms()
        {
            return await serializer.Deserialize();
        }

        /// <summary>
        /// Adds a persisted alarm.
        /// </summary>
        private static async Task AddPersistedAlarm(Alarm alarm)
        {
            var alarms = await PersistedAlarms();

            await RemovePersistedAlarm(alarm);

            alarm.TimeSpan = alarm.TimeLeft;
            alarms.Add(alarm);

            await serializer.Serialize(alarms);
        }

I reused the XML Serializer from this article. Because the native XML Serializer doesn’t handle the TimeSpan data type very well, I applied a little trick to serialize and deserialize the TimeSpan property. I created (serializable) shadow property that holds the number of Ticks in it:

        /// <summary>
        /// Gets or sets the time span.
        /// </summary>
        /// <remarks>Not XML serializable.</remarks>
        [XmlIgnore]
        public TimeSpan TimeSpan
        {
            get { return this.timeSpan; }
            set { this.SetProperty(ref this.timeSpan, value); }
        }

        /// <summary>
        /// Gets or sets the time span ticks.
        /// </summary>
        /// <remarks>Pretended property for serialization</remarks>
        [XmlElement("TimeSpan")]
        public long TimeSpanTicks
        {
            get { return this.timeSpan.Ticks; }
            set { this.timeSpan = new TimeSpan(value); }
        }

When an individual alarm is disabled, then we unschedule the corresponding toast notification, and remove it from the list of persisted alarms:

        /// <summary>
        /// Disables this instance.
        /// </summary>
        public async Task Disable()
        {
            if (this.state != AlarmStates.Scheduled && this.state != AlarmStates.Paused)
            {
                // No action.
                return;
            }

            this.UnscheduleToast();
            this.State = AlarmStates.Canceled;

            await RemovePersistedAlarm(this);
        }
        /// <summary>
        /// Unschedules the toast.
        /// </summary>
        private void UnscheduleToast()
        {
            var toasts = toastNotifier.GetScheduledToastNotifications();
            var found = (from t in toasts
                         where t.Id == this.Id
                         select t).FirstOrDefault();
            if (found != null)
            {
                toastNotifier.RemoveFromSchedule(found);
            }
        }
        /// <summary>
        /// Removes a persisted alarm.
        /// </summary>
        private static async Task RemovePersistedAlarm(Alarm alarm)
        {
            var alarms = await PersistedAlarms();

            alarms.RemoveAll(a => a.Id == alarm.Id);

            await serializer.Serialize(alarms);
        }

When the user starts the app by tapping on the tile, or on a toast, the app reassembles the (remaining) workflow by combining the scheduled toast notifications with the serialized paused alarms (we deliberately forget the steps for which the alarms were delivered). So this is how the app looks like after a restart:

Reopen

This is the code that brings back the list of alarms, it comes from the constructor of the main view model – which actually represents the workflow:

            // Create an alarm for all scheduled notifications from previous sessions that are still running.
            var scheduled = Alarm.ScheduledAlarms();

            if (scheduled.Count() > 0)
            {
                foreach (var alarm in scheduled)
                {
                    this.alarms.Add(alarm);
                }
            }

            CoreWindow.GetForCurrentThread().Dispatcher.RunAsync
                (
                    CoreDispatcherPriority.Normal,
                    async () =>
                    {
                        // Rehydrate paused alarms.
                        var persisted = await Alarm.PersistedAlarms();
                        if (persisted.Count() > 0)
                        {
                            foreach (var alarm in persisted)
                            {
                                this.alarms.Add(alarm);
                            }
                        }

                        // Create default alarms.
                        if (this.alarms.Count == 0)
                        {
                            Toast.ShowInfo("Welcome", "I created some default alarms for you.");
                            this.CreateDefaultAlarms();
                        }

                        this.RescheduleTiles();
                    }
                );

            DispatcherTimer toastTimer = new DispatcherTimer();
            toastTimer.Tick += this.ToastTimer_Tick;
            toastTimer.Interval = TimeSpan.FromSeconds(1);
            toastTimer.Start();

At the end of the previous code snippet, you see that we fire up a timer with a short interval (1 second). This timer updates the UI so the user has a detailed view on the workflow status through the changing TimeLeft fields. But it doesn’t stop here: we also want to give the user a high level –but less accurate- overview of the status of the running workflow. Therefor we decorate the app’s live tile with a message that contains the remaining time to the next notification and the remaining time for the entire flow. For that we schedule a regular tile update; every minute for the sample app, but I can imagine you would want a larger interval for a production app. Here’s the code to schedule these tile notifications, it’s very similar to the toast scheduling. This code is executed when the app restarts, when user modifies the flow, and also on a slow(er) moving timer.

For each minute between the current moment and the end of the workflow we lookup the next upcoming toast, and create the corresponding title and message on the live tile. The sample app notifies only the TimeLeft values, but you have access to all properties of the associated alarms (including Title and Content) if you want. Just remember that you only have three short lines of text for the tile:

        /// <summary>
        /// Reschedules the tiles.
        /// </summary>
        private void RescheduleTiles()
        {
            var scheduledAlarms = this.ScheduledAlarms;

            var nextAlarm = scheduledAlarms.FirstOrDefault();
            var lastAlarm = scheduledAlarms.LastOrDefault();

            if (nextAlarm == null)
            {
                // No alarms
                UnscheduleTileNotifications();
                return;
            }

            var next = (DateTime.Now + nextAlarm.TimeLeft) - nextAlarmTime;
            var nextMinutes = next.TotalMinutes;

            var last = (DateTime.Now + lastAlarm.TimeLeft) - lastAlarmTime;
            var lastMinutes = last.TotalMinutes;

            if ((Math.Abs(nextMinutes) < 2) && Math.Abs(lastMinutes) < 2)
            {
                // Nothing changed since the previous check.
                return;
            }

            nextAlarmTime = DateTime.Now.Add(nextAlarm.TimeLeft);
            lastAlarmTime = DateTime.Now.Add(lastAlarm.TimeLeft);

            UnscheduleTileNotifications();

            DateTime dateTime = DateTime.Now.AddSeconds(5);
            while (dateTime < lastAlarmTime)
            {
                string title = "Cooking";

                var alarm = (from a in scheduledAlarms
                             where a.TimeLeft > dateTime - DateTime.Now
                             select a).FirstOrDefault();

                if (alarm != null)
                {
                    string content;

                    if (alarm != lastAlarm)
                    {
                        content = String.Format(
                             "Notifies in {0} min.\nEnds in {1} min.",
                             (alarm.TimeLeft - (dateTime - DateTime.Now)).Minutes + 1,
                             (lastAlarm.TimeLeft - (dateTime - DateTime.Now)).Minutes + 1);
                    }
                    else
                    {
                        content = String.Format(
                             "Ends in {0} min.",
                             (lastAlarm.TimeLeft - (dateTime - DateTime.Now)).Minutes + 1);
                    }

                    tileUpdater.AddToSchedule(new ScheduledTileNotification(this.GetTile(title, content), dateTime));
                    Debug.WriteLine("Scheduled Tile Notification for {0}.", dateTime);
                }

                dateTime = dateTime.Add(TimeSpan.FromMinutes(1));
            }

            // Done.
            tileUpdater.AddToSchedule(new ScheduledTileNotification(this.GetTile("Job done", string.Empty), dateTime));
        }

Here’s an MS-Paintoshopped image of the evolution of the sample app's live tile throughout the workflow:

TileEvolution

The ‘Job Done’ tile notification is the last one. The tile remains like this until the app is restarted, since there seems to be no way to bring back the default tile, at least not through the scheduling infrastructure. If you really need this functionality, you could start a background job. Anyway, here’s the call to bring back the default tile:

//Reset to default tile.
tileUpdater.Clear();

The sample app uses a strictly sequential workflow, but the Alarm ViewModel class is reusable in more complex scenario’s with e.g. parallel tasks or tasks that can be paused individually.

Here’s the full source code, it was built with Visual Studio 2013 Update 2: U2UC.Win8.Alarms.zip (314.3KB)

Enjoy!

Diederik

Building Enterprise Side-loaded apps with OWIN

This article shows how a side-loaded line-of-business Windows app can communicate through HTTP with a locally installed desktop component (a .NET class library) by making use of the Open Web Interface for .NET. Apps that are loaded from the Store can still not leave the sandbox and can only connect to servers, but since Windows 8.1 side-loaded Enterprise apps are allowed to open ports in the firewall around their own app container. Such apps are now allowed e.g. to connect to http://localhost. For more info, just check the first half of this Build 2014 session.

I can imagine that a lot of companies would like to roll out some Modern LOB apps that talk to locally installed components. HTTP is a nice protocol for that, and ASP.NET Web API would make a nice host to expose local components. If that’s your idea too, then here’s the bad news: most network administrators will simply not allow the installation/activation of an infrastructure such as IIS on the regular workstation of each employee. But don’t worry, here’s the good news: among a lot of other capabilities, the Open Web Interface for .NET (OWIN) allows you to self-host a lightweight web server in a Windows service or a console application without the need for IIS.

To illustrate this configuration, I built a small app that talks to a localhost REST provider in a .NET console app which in its turn accesses a local “legacy” .NET class library. Here’s how the app looks like:

AppWithOwin

The resemblance of this app with the one from my previous blog post is not a coincidence, of course; and I’m also reusing the legacy projects. I just added an extra button to the Windows Forms application to test the new OWIN call:

FormsClient

This particular use of OWIN has the same purpose as Brokered Windows Runtime Components: connecting Modern touch-optimized apps to software components local to the desktop.

Here we go, let’s start with a self-hosted Web API server. In a production environment we would do this in a Windows Service, but for now just create a new C# Windows console, no special template needed:

ConsoleApp

It may add too many references, but the fastest way to take off with OWIN as a Web API host, is installing the ASP.NET Web API OWIN Self Host Nuget package:

OwinNuget

Create a StartUp class to configure the host. If you ever encountered ASP.NET MVC or Web API, then it should look familiar:

namespace U2UC.WinLob.OwinHost
{
    using Owin;
    using System.Web.Http;

    public class Startup
    {
        // This code configures Web API. The Startup class is specified as a type
        // parameter in the WebApp.Start method.
        public void Configuration(IAppBuilder appBuilder)
        {
            // Configure Web API for self-host. 
            HttpConfiguration config = new HttpConfiguration();
            config.Routes.MapHttpRoute(
                name: "DefaultApi",
                routeTemplate: "api/{controller}/{id}",
                defaults: new { id = RouteParameter.Optional }
            );

            appBuilder.UseWebApi(config);
        }
    }
}

Then add a Web API controller that wraps the legacy component (add a reference) in a REST interface. The sample app will only use a GET call to retrieve the list of Northwind employees (as a JSON array):

namespace U2UC.WinLob.OwinHost
{
    using System;
    using System.Collections.Generic;
    using System.Web.Http;

    public class NorthwindController : ApiController
    {
        public IEnumerable<Employee> Get()
        {
            List<Employee> result = new List<Employee>();

            foreach (var emp in U2UC.WinLob.Legacy.Dal.GetAllEmployees())
            {
                result.Add(
                    new Employee() 
                        {
                            Name = emp.Name,
                            Title = emp.Title,
                            BirthDate = emp.BirthDate,
                            Picture = Convert.ToBase64String(emp.Picture)
                        }
                    );
            }

            return result;
        }
    }
}

The last step is to start up the OWIN host, that’s done in the console’s Main routine. The test call is not really necessary, but it’s there to show us that the host is operational:

namespace U2UC.WinLob.OwinHost
{
    using Microsoft.Owin.Hosting;
    using Newtonsoft.Json;
    using System;
    using System.Collections.Generic;
    using System.Net.Http;

    // More details on hosting WebApi here:
    // http://www.asp.net/web-api/overview/hosting-aspnet-web-api/use-owin-to-self-host-web-api
    public class Program
    {
        static void Main()
        {
            string baseAddress = "http://localhost:9000/";

            // Start OWIN host 
            using (WebApp.Start<Startup>(url: baseAddress))
            {
                // Create HttpCient and make a request to api/northwind 
                HttpClient client = new HttpClient();
                var response = client.GetStringAsync(baseAddress + "api/northwind").Result;

                List<Employee> employees = JsonConvert.DeserializeObject<List<Employee>>(response);

                Console.WriteLine("{0} Northwind employees found.", employees.Count);

                Console.ReadLine(); // In the using, not after it like in the demo :-)
            }
        }
    }
}

When the console app is started, it looks like this:

TestHost

The console app is now exposing the functionality of the legacy library through HTTP, REST, and JSON.

Let’s take a look at the side-loaded Modern app now. It helps to make a reference to JSON.NET:

JsonApp

When the app needs to call the legacy code, it creates a HttpClient, executes the asynchronous call, and parses the JSON result. By the way: JSON.NET comes with excellent LINQ capabilities for that last step. The following code snippet comes from the code-behind of the page. I'm not using MVVM here, shame on me:

private void AppBarButton_Click(object sender, RoutedEventArgs e)
{
    // Create HttpCient and make a request to api/northwind 
    var client = new HttpClient();
    var json = client.GetStringAsync(new Uri("http://localhost:9000/api/northwind")).Result;
            
    var employees = JsonConvert.DeserializeObject<List<Employee>>(json);

    this.Employees.ItemsSource = employees;
}

That’s it: the app now accesses local legacy code using only standard protocols. I’m missing something in the varbinary-bytearray-base64string-bitmapimage conversion of the database pictures, but it’s still a representative demo :-)

Here’s the full sample app, it was built with Visual Studio 2013. Just remember to define the App and the OwinHost as startup projects : U2UC.WinLob.Owin.zip (8MB)

Enjoy!

XAML Brewer