U2U Blog

for developers and other creative minds

AngularJS: Async Callbacks and the Angular Execution Context

I was playing around with a Angular and Google Maps, and suddenly the world fell apart. Ok, maybe I'm a bit dramatic here, but angular behaved quite a bit different than I expected.

This is a simplified version of the code that cracked the foundations of my beloved planet:

$scope.geoCode = function () {

  $scope.geocoder.geocode({ 'address': $scope.data.search }, function (results, status) {
    var loc = results[0].geometry.location;
    $scope.data.search = results[0].formatted_address;
    $scope.data.location = { lat: loc.lat(), lon: loc.lng() };
  });
};

This piece of code is part of a controller, when clicking a button it searches for a address and a geolocation based upon the search query.

Now the surprising thing here is that I had to press that button twice to make it work. And here's why: only code that executes in the Angular Execution Context is being watched. Angular does dirty checking to inform the watchers (the ones that call $watch) that something has changed, so the watcher can do things like re-render, re-calculate, etc.

We can explicitly enter the Angular Execution Context(AEC) by calling $apply(fn), but most of the time this is not necessary since Angular calls that behind the scene for us. And this is what's causing the problem. Angular executes the code in my controller in the AEC, but the asynchronous callback is not, and Angular remains unaware of any changes.

We have found the poison, but we also have the remedy, right? Just call $apply!

$scope.geoCode = function () {

  $scope.geocoder.geocode({ 'address': $scope.data.search }, function (results, status) {
    var loc = results[0].geometry.location;
    $scope.data.search = results[0].formatted_address;
    $scope.data.location = { lat: loc.lat(), lon: loc.lng() };
    $scope.$apply("data.location");
  });
}; 

Super! except that now, it works half of the time. And the other half, well..., it sends me this little message:

image

It seems that the $digest loop was still ongoing and you can not call $apply as long as it is running. How long the $digest loop takes is unpredictable. As unpredictable like world-shaking earthquakes! When a model’s value changes, the watchers may respond with even more changes. The $digest loops continues until the model(s) stabilize. For more information about $digest click here.

Hold on! I'm probably not the first guy to ever use an async callback in a controller, what about web service calls for example. Well what about them? Here is an example:

$http.get(url).
  success(function (data) {
    $scope.model.country = data.geonames[0].countryName;
  }).
  error(function (data, status) {
    $scope.model.country = 'server answered with: ' + status;
  });

And this always works. Apparently these callbacks are always called in the AEC without the risk of a re-entry of the $apply. Let's dive into the library itself, and see how it's done.

After scuba diving through the internals of angular for a while I found this little gem:

if (!$rootScope.$$phase) $rootScope.$apply();

It seems that this $$phase flag can hold on of three values, either '$apply', '$digest' or undefined. If undefined, the digest loop has already stopped and we can safely call $apply(). If the digest loop is still ongoing, then my added change to the scope will cause a de-stabilization in the model, and the current digest phase will process my change.

Here is an updated version of my code:

$scope.geoCode = function () {

  $scope.geocoder.geocode({ 'address': $scope.data.search }, function (results, status) {
    var loc = results[0].geometry.location;
    $scope.data.search = results[0].formatted_address;
    $scope.data.location = { lat: loc.lat(), lon: loc.lng() };
    if (!$scope.$$phase) $scope.$apply("data.location");
  });
};

And guess what, it works!

The rumbling sound of colliding rocks grow silent and all is well. Until I suddenly realized that this might not be the best solution ever. $$phase is nowhere to be found in the documentation, and there might be a good reason for that. Also going into the internals of a library is kind of against the rules of encapsulation. Angular might decide to change its internals. As long as the API stays the same, nothing can prevent them. (except for an earthquake, that can stop anything)

So, one final attempt:

$scope.geoCode = function () {

  $scope.geocoder.geocode({ 'address': $scope.data.search }, function (results, status) {

    $timeout(function () {
      var loc = results[0].geometry.location;
      $scope.data.search = results[0].formatted_address;
      $scope.data.location = { lat: loc.lat(), lon: loc.lng() };
    });

  });
};

Basically I'm high jacking the $timeout service. The callback always executes nicely in the AEC and I don't have to get into the guts of Angular.

The only problem with this last one is that it might not be clear to another developer why someone would wrap this callback into a $timeout call. It's just something that you have to know.

Well now you know, and knowing is half the battle. If you want to learn more about AngularJS check out our course.

Here is something to look at while letting it sink:

quake

Challenges in combining multiple data sets in PowerBI

Relationship between house price and income

A few days ago the media reported that Belgium was amongst one of the most expensive European countries to buy a house, if you compare the house price relative to the income. A few days ago I wrote a blog post on how we could analyze the Belgian house market using Microsoft Excel PowerBI and public data from the Belgian federal government (http://statbel.fgov.be). Inspired by the news that the house price relative to the income is an important indicator, let’s try to mimick this research, but then specifically for the Belgian house market.

With PowerPivot one cannot only model data coming from a single data source, it is also easy to combine different data sources into a single model. For this blog post, I took the model made in the previous Belgian house market analysis (with house prices per city) and then looked for a dataset with income information per city.

On http://statbel.fgov.be/nl/modules/publications/statistiques/arbeidsmarkt_levensomstandigheden/Fiscale_inkomens_-_per_gemeente_-_2008.jsp we can get fiscal income per Belgian city in 2008 (unfortunately I’m not aware of more recent public datasets on this). Using PowerQuery we can filter and convert the data such that we the total fiscal income, number of taxpayers and number of inhabitants per city.

Combining the data sets: a risky business

After we loaded this data in the model, we need to setup the proper relationships between the two data sources. Since both data sources are using the same city identifier (NIScode), this link is easily made. However, if we would directly link the table with the house prices per city (which contains a row per city per year per house type per measurement) with the table with the incomes per city (which contains a row per city per year), we can get errors. PowerPivot does not allow many-to-many relationships! In our model it is even worse. Since we currently only loaded data for 2008, we do only have one row per city in the income table. But as soon as we load data for multiple years, our model would become invalid!

image

The proper way to link data together

Even with a small data set (just 2 tables) as we have in this example, it is already important to stick to some of the core ideas in business intelligence: dimensional modeling. This boils down to splitting up data into dimensions (things you group or filter upon, typically the nouns in the business story) and facts (things you aggregate upon in your reports, typically the verbs in the business story). In our analysis we have two dimensions: Cities and years. So we must create a table with only one row per city (with all the information regarding that city: code, name, county, country), and another with one row per year. Since there is not much more to tell about a year than just its value, we can easily generate the latter table in PowerQuery:

   1: let
   2:     Source = List.Numbers(1990,25),
   3:     TableFromList = Table.FromList(Source, Splitter.SplitByNothing(), {"Year"}, null, ExtraValues.Error)
   4: in
   5:     TableFromList

After these tables get created, we can now link them in PowerPivot in this way:
image

From the perspective of linking the tables, the solution is fine. But we still run the risk that users will for instance grab the Years or NIScode column from either the HousePrice or the income table in one of their reports. This would be problematic, since this would only filter data from the table from which it was choosen, not from the other table. However, if we can force our users to select years only from the Years table and NIScodes only from the Location table, we don’t face this problem, since both our fact tables (HousePrice and Income) are related to both these dimension tables (Years and Location). This can easily be obtained by just hiding these columns in the fact tables:
image

After we made the model, we can now start defining DAX measures which are calculated over both fact tables. A very interesting one is the average house price expressed in terms of the average income: How many years of average income in a city do people need to spend to buy an average house in that city.

Why Bruges isn’t the most expensive area after all

Now we can start building PowerView reports on top of this data. In this way we can for instance see that Bruges is much more expensive than Brussels, but if we express the houseprice in terms of the income of its inhabitants, Bruges becomes cheaper than Brussels:
image

Download the Excel sheet from my OneDrive, and start building lots of interesting reports on top of this data set! Check out our Excel PowerBI course to learn more about building models and reports in Excel.

Analyze the Belgian house market… yourself

From time to time articles get published about the evolution of the Belgian house market. Interesting to learn about general trends but… sometimes you whished these analyses would be more tailored to your situation. It’s nice to know that prices for villas have stabilized last year, but does this trend also hold in my region? Sometimes we just have to do things our self…

PowerBI = Self Service BI

Lot’s of data is nowadays publicly available. The same holds for the Belgian house market analysis: http://statbel.fgov.be has a lot of public data sets available for analyzing different aspects of Belgium. One of them contains house sales information per village per year per house type (flat, house, villa). By diving into this data set we can find all sorts of detailed information.
image

But its not easy to navigate this data. We want to massage the data such that it becomes easier to filter and aggregate the data. To do so we can use many tools… but why leave Excel? In 2014 Microsoft released PowerBI, which amongst others contains Excel plugins for loading, cleaning and modeling data.

PowerQuery makes data loading easy

With the free PowerQuery plugin, we can easily create a script which downloads the data from the statbel website, filters and transforms the data until we become a representation which is easier to filter and aggregate:
image

If you want to give it a try, here is the M code, just copy it and paste it into a blank PowerQuery.

   1: let
   2:     Source = Excel.Workbook(Web.Contents("http://statbel.fgov.be/nl/binaries/NL_immo_statbel_jaar_131202_%20121715_tcm325-34189.xls")),
   3:     #"Per gemeente1" = Source{[Name="Per gemeente"]}[Data],
   4:     RemovedFirstRows = Table.Skip(#"Per gemeente1",2),
   5:     FirstRowAsHeader = Table.PromoteHeaders(RemovedFirstRows),
   6:     RemovedColumns = Table.RemoveColumns(FirstRowAsHeader,{"localiteit", "jaar_1", "oppervlakteklasse", "Column6", "P10 prijs(€)", "Q25 prijs(€)", "Q50 prijs(€)", "Q75 prijs(€)", "P90 prijs(€)", "Column16", "P10 prijs(€)_6", "Q25 prijs(€)_7", "Q50 prijs(€)_8", "Q75 prijs(€)_9", "P90 prijs(€)_10", "Column26", "P10 prijs(€)_15", "Q25 prijs(€)_16", "Q50 prijs(€)_17", "Q75 prijs(€)_18", "P90 prijs(€)_19", "Column36", "P10 prijs(€/m²)", "Q25 prijs(€/m²)", "Q50 prijs(€/m²)", "Q75 prijs(€/m²)", "P90 prijs(€/m²)", "gemiddelde prijs(€)", "gemiddelde prijs(€)_5", "gemiddelde prijs(€)_14", "gemiddelde prijs(€/m²)"}),
   7:     RenamedColumns = Table.RenameColumns(RemovedColumns,{{"aantal transacties", "House transaction count"}, {"totale prijs(€)", "House total cost"}, {"totale oppervlakte(m²)", "House building area"}, {"aantal transacties_2", "Villa transaction count"}, {"totale prijs(€)_3", "Villa total cost"}, {"totale oppervlakte(m²)_4", "Villa building area"}, {"aantal transacties_11", "Flat transaction count"}, {"totale prijs(€)_12", "Flat total cost"}}),
   8:     RemovedColumns1 = Table.RemoveColumns(RenamedColumns,{"totale oppervlakte(m²)_13"}),
   9:     RenamedColumns1 = Table.RenameColumns(RemovedColumns1,{ {"aantal transacties_20", "Ground transaction count"}, {"totale prijs(€)_21", "Ground total cost"}, {"totale oppervlakte(m²)_22", "Ground building area"}}),
  10:     Unpivot = Table.UnpivotOtherColumns(RenamedColumns1,{"refnis", "jaar"},"Attribute","Value"),
  11:     SplitColumnDelimiter = Table.SplitColumn(Unpivot,"Attribute",Splitter.SplitTextByEachDelimiter({" "}, null, false),{"Attribute.1", "Attribute.2"}),
  12:     ChangedType = Table.TransformColumnTypes(SplitColumnDelimiter,{{"Attribute.1", type text}, {"Attribute.2", type text}}),
  13:     RenamedColumns2 = Table.RenameColumns(ChangedType,{{"Attribute.1", "HouseType"}, {"Attribute.2", "Measurement"}}),
  14:     ChangedType1 = Table.TransformColumnTypes(RenamedColumns2,{{"Value", type number}}),
  15:     TransformedColumn = Table.TransformColumns(ChangedType1,{{"Measurement", Text.Proper}}),
  16:     RenamedColumns3 = Table.RenameColumns(TransformedColumn,{{"jaar", "Year"}}),
  17:     ChangedType2 = Table.TransformColumnTypes(RenamedColumns3,{{"Year", type date}}),
  18:     RenamedColumns4 = Table.RenameColumns(ChangedType2,{{"refnis", "NISCode"}})
  19: in
  20:     RenamedColumns4

PowerPivot to model the data the way you want it

Next we want to define useful calculations: Sums, averages, but also more challenging calculations, such as linking each village with the corresponding county. For this we use PowerPivot, a free addin for Excel 2010, out-of-the-box present in Excel 2013. Amongst others we define an average cost per house and an average cost per square meter calculation:
image

PowerView to build interactive charts

As soon as we have loaded and modeled our data, we can finally start building the reports we needed. The PowerView add-in in Excel 2013 allows us to quickly create interactive reports on top of our PowerPivot model. In less than two minutes we can create a report which shows the price evolution over time for the different counties, with a drill down into the actual villages, split by house type. Watch the video below to get an idea, or if you have Excel 2013, just download the Excel file and play with the report itself!

 

The nice thing about PowerView is that out-of-the-box it usually does what we expect it to do. For instance: when we click on a house type, the bar charts automatically show the average price for the selected type of house relative to the average price for all house types. For instance, in the county Brugge a flat is more expensive than the average price for any type of house, whereas in the other counties it is cheaper:
image 

We can also create more advanced visualization. For instance a bubble chart plotting the average land price against the average size of the building area, and how this evolves over time. When you single click a county, you see the price evolution through time, double clicking will zoom in into the cities within that county.

Do it yourself

You can download the Excel sheet from my OneDrive at http://1drv.ms/1fcH2u6 and have fun with the data. Be sure to download the Excel file, since OneDrive does not allow you to interact with the data in the browser.

But if you want to learn how to build PowerPivot models and load data with PowerQuery, consider attending our three day course Building PowerPivot Models in Excel. If you are also interested in building PowerView and PowerMap reports on top of the PowerPivot models, you can instead attend the course Business Intelligence with Excel.

Have fun with the PowerBI self service business intelligence… the world is yours to discover!