Currently the Azure Digital Twin Service is still in preview. There are some interesting facts and small issues you need to know before starting to try this out. To understand what Azure Digital Twin offers, it would be best to read my extended post about an introduction to Microsoft Digital Twin. So let’s start!
Availability of Digital Twin Preview
Or the lack of it. You are lucky if you already had a created Azure resource based on the Digital Twin service. At the moment Microsoft states :
“Thank you for your interest in the Azure Digital Twins preview program. Due to overwhelming demand, the preview program is temporarily closed as we prepare for the upcoming release of new capabilities. As a result, you may not be able to create new Azure Digital Twins resources right now. Please continue checking back for new information.“
So be careful not to throw your resource away. A good chance is that you are not able to create one again. I tried several locations worldwide without luck. Luckily I already had one.
Microsoft has an interesting example which allows you to play around with the Azure Digital Twin service without having an actual device. It uses a separate project which communicates through the IoT hub to provide the Digital Twin simulated data.
IoT Hub… where is it?
Talking about IoT Hub. Wonder where that one is? It is there, but you cannot access it. It is automatically generated when the Azure Digital Twin was created. It will not appear in the list of resources in your Azure environment. Even worse… you can’t access it via Azure Powershell.
Digital Twin viewer
There is a great Digital Twin viewer. But you will get into an issue, when you have done the example before the viewer. The redirect URL of the example is the same as the viewer which causes issues by logging into Azure via the browser using the example. The best way is to change the port number of the redirect URL of the project. Add an additional URL to the App registration under the Azure Active Directory to support this.
There are several limitations using the Azure Digital Twin Preview. You are allowed to run only one Azure Digital Twin service at the time. There are rate and message limitations for Azure Digital Twin Management API, user-defined functions and device telemetry. Read about them here.
Playing around with Azure Digital Twin Preview is just great! As long as you understand the limitations you can try several things yourself without having an actual IoT device. It gives you a good understanding and ideas where you can use it for. We need to wait for general availability to start using it in businesses. There are some other interesting findings posted in the feedback hub which you can find here.
https://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.png00Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2020-05-21 15:10:232020-05-26 21:22:00Interesting facts about Azure Digital Twin Service Preview
It is more than 4 years ago that Microsoft came with the first release of Microsoft HoloLens device in the world of Mixed Reality. It took some time before Microsoft released several out-of-the-box applications for businesses. And they are ruling the Mixed Reality landscape since then. Their focus stays continuously on providing solutions which addresses the most common challenges for businesses.
One of these business challenges is training personnel. Training personnel in organizations require a lot of effort from experts on the workforce. And with a large staff turnover, something which happens a lot on the factory floor, is training a costly part of their business. But even keeping your workforce up-to-date with the latest requirements for training, or to prep students on school for what is coming or getting more insights by combining it with for example a Digital Twin will improve, support and saving costs for your organization allowing you to drive better business.
It is important for organizations to understand the business value of Microsoft Guides. Eventually organizations want to empower their workforce, optimize digital operations and deliver new services internally or externally to their customers. And that will result in a different way of thinking within manufacturing, training, support and guidance of employees.
Microsoft Guides is a Mixed Reality business application which allows you to create a tailor made guide for employees on the factory floor. It allows you to create holographic work instructions which support a whole or part of the work flow. There are absolutely no coding skills requires for creating these Mixed Reality guides.
Microsoft Guides is part of the Microsoft Dynamics 365 family and therefor actually called Microsoft Dynamics 365 Guides. It uses the base storage system of Dynamics 365, its underlying Common Data Service and Power Apps to store its flow and content.
The solution works for Microsoft HoloLens 1 and 2. But understandable it is extended based on the new functionality provided by Microsoft HoloLens 2 like natural gestures and other. The natural gestures are currently only used for the author when building guide.
Since then the Microsoft Team responsible for Microsoft Guides is extending the functionality of the application and its platform on a monthly basis. They are keen on getting feedback from customers, MVP’s and consultants. And they take the feedback seriously.
How does it work?
In short a guide is created by an author. The author starts outlining a guide into different tasks using a desktop application. Outlining is based on requirements of the customer and the guidelines of building a guide. Then the author switches to the Microsoft HoloLens 2 device. There the guide is attached to for example an object in the real physical world by editing each task of the outlines. This is accomplished by moving, placing, sizing and rotating the holograms around in the real physical world. Adding additional holographic instructions like arrows, hands and more.
An operator logs in Microsoft Guides and select the guide. The operator needs to synchronize his training using a tag or positioning a holographic object on the object in the real world. After this the guide is executed through the outlines and tasks. The operator experiences the cards with the steps explained floating around. A tether (dotted line) is used to indicate where the step needs to be executed on the real assets. Finally the author is able to monitor and analyse the progress and execution. It allows the author to improve the guides and support the operator in improving their skills.
Microsoft Guides allows you to use different approaches.
Real-life assets – This is the best way of having hands-on training for employees. The guide is synchronized on a real-life object using a tag or holographic object.
Virtual assets – guides with virtual assets allow you to have education or training without the actual real-life asset. The asset is digitally placed as a holographic object in each task. This is specifically interesting in situations where you are not able to have the actual asset. It is also a more passive way of training since you are not able to execute the tasks on the real-life asset.
Before you start
One of the biggest mistakes is thinking that creating a guide is simply creating a few outlines and task. It is much more comprehensive. Before you start you will need to gather as much content as possible to create your guide. This is accomplished by having for example workshops and inspiration sessions with your customer or target group. There are a few things which are important for creating your guide. You need to understand the space where the guide is build for. You need to understand the procedure and workflow of the actual work. These things will influence in how you build your guide. If possible try to involve an expert of the organization. Actually it is a must to create a good guide. Keep that in mind!
Also try to understand the objectives and sub-objectives of your guide. This will influence the amount of outlines and tasks you want to use. You would think why? But it is important to not lose your operator using the training into doing too much tasks within the same outline. Something we will explain further in detail. And the final one is taking into consideration for which role you are creating a training. The role determines what you need to highlight from the asset and the depth of information you want to provide.
Guides makes use of an anchor. The anchor is used to position the guide at the right location in the real world. This is accomplished by using a QR code or a Circular code. The later one is a predefined tag by the Microsoft Guides team. It is also possible to use a holographic object as a marker. But that requires to position the holographic object at the exact position which is in most cases very difficult. This is mostly interesting when it is not possible to place a tag on the actual asset.
Structure of a guide
The structure of guides exists of one or more cards placed in an outline. Your guide can have more than one outline. the outlines and cards are executed in a logical sequence. There are rules for creating fabulous and great working guides. Information about making guides great can be found here. Not going too much in detail, but it explains from begin to end and has a large number of great tips and tricks. Examples are that each outline must be a specific task within the work flow. It should have a clear beginning and end. It explains how to get around with holographic objects and to be consistent in using models, styles, texts and more.
The outlines and tasks are created via the desktop application. The desktop application allows you to add content. Content can be anything like images, sounds, video and holographic models. It offers out of the box several holographic helpful parts like arrows, generic tools, hands, numbers, symbols and animated zones.
A really cool feature which is just added two months ago is having actions in a card. At the moment there are only two actions. But i would expect more in the future. And who knows… maybe even add your own actions. But for now we have the action website link an Power App.
The website link allows you to add one specific link per card which can be opened during the presentation of the card. Imagine using a link to more information about what the employee is doing. Or maybe real-time information coming from a sensory device shown at the real object or in the virtual model.
Power Apps is a major cool feature. It allows you to show a Power App during a card in the outline. Just like the website link you can have a different Power App per card. Imagine a Power App with questions to be answered by the employee. You could for example use a single Power App with several questions. Each question can be reached by using a parameter in the URL to the Power App. And the results are stored in a database based on the credentials of the logged on user. There are so many more things you could do with this. And just like Microsoft Guides, Power Apps has a creator interface for power users. No additional coding and technical skills are required.
Roles and rights
At the moment Microsoft Guides knows two roles. We have the author and the operator.
The author is allowed to create, rename and edit guides. The author can also activate or deactivate guides. Just like the operator the author can also operate guides.
The operator is allowed to view and operate assigned guides via the Microsoft HoloLens.
Improve efficiency by analytics
The performance of the operator is measured due to data collection during the operation of a guide. Each gaze and commit interaction on buttons is measured. Time related information about the run, guide task and step are stored. This allows the author to view usage statistics and detailed time-track information via Power BI reports. These metrics can be used by the author to optimize the created guide. But it can also be used to see how the operator’s performance is improving doing the work. Sharing the result using Power BI Reports requires an appropriate license.
There are two default reports available. The process time-tracking report and the Guides usage report. These reports can answer several questions like
Is daily guide usage changing?
What is the most frequently used guide?
What is the average run time per guide?
How long is a guide run in minutes?
How long did each task or step take?
Power Apps & Power automate
Microsoft Guides can be integrated into existing process and workflows using Power Apps and Flow. You can start a workflow or even use the “Create work record” event. This integration is mainly on the back-end of Guides and differs therefor on the Power App action discussed earlier.
Dynamics 365 Field Services
There is also an integration with Dynamics 365 Field services. It allows you to attach guides to Field Services tasks. This gives you the ability to complete work orders via Microsoft Guides.
Microsoft Guides is the tool for building great guides in Mixed Reality. While the tool is just released last year, it offers an extensive amount of functionality which is extended every month. Dynamics 365 and Power Apps are the base for Microsoft Guides. And do not get mistaken. Power Apps is going to be big and very important through the Microsoft landscape. The same is for Microsoft Guides as tool to build guides and more.
https://globalxr.community/wp-content/uploads/2020/05/businessvalueofguides-1024x568-1.png5681024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2020-05-18 09:56:112020-05-26 21:22:00Microsoft Guides, the next step in training & guidance with Mixed Reality
Digital Twin? That’s a buzzword for something isn’t it? Could be. But it is actually an interesting type of service which allows you as an IT company explain a business solution to your customers without going into depth about Machine Learning, Mixed Reality, Internet of Things, Sensory data and many other technologies. It is a business solution which can easily solve several use cases for your customer. And they aren’t particularly interested in what technologies you would need to use, as long as it does the job.
What is a Digital Twin?
So what is a Digital Twin? A Digital Twin is a digital representation, also called a copy, of physical entity(s) in the real world. A physical entity can be anything from people, places, machines, factories, devices, systems and even processes. One of the characteristics of a Digital Twin is that the replica represents a, if possible a true, copy of one or more actual physical entities. Such a copy is visualized using a 2D form like a web browser or a 3D form using for example a Microsoft HoloLens device or mobile device supporting augmented reality. These physical entities generate real-time data which is feed back into the model of the Digital Twin. In some cases a Digital Twin is compared to concepts like cross-reality environments and mirror models.
The idea is creating a “living” simulation model. The model continuous update and change based on the actual values of real–life entities. Such a simulation model can be used in several different use-cases. Use cases for example
Simulation – Simulate different scenarios which you normally never would do an the actual asset without damaging it or causing collateral damages.
Analyse & Optimize – optimize systems, machines or processes by tweaking values in the simulation model. This allows you to see the effect on those entities without changing the actual configuration. When a better and more optimized configuration is found it can be applied to the actual real-life entities.
Training – simulation models allow you to train new personnel without using the actual machine or control room. Some scenarios You could also use it in education at schools
Testing – use the simulation model to find the utter edges of the system before it fails.
Visualization – get more and clearer insights in current statuses of systems and machine over multiple locations spread.
Complex models – Bring complex scenarios with lots of sensory devices divided over structured locations into a single model. Combine and use the data to create easy and simple to understand views.
Professional services – Use sensory data from devices at different locations to provide services to your company. The Microsoft example which is available for Digital Twin is a great example. In this example different types of sensory data, temperature, movement and other, is used to determine if a room is available and suited for you.
You can understand that there are far more examples in which a Digital Twin can offer benefits.
Microsoft Azure Digital Twin Service
The Azure Digital Twin service is an IoT service which helps you in creating models of physical environments. It uses something called a spatial intelligence graph. The spatial intelligence graph allows you to model a structure of relationships and interactions between people, places and devices. This allows you to query real-time data from devices which are bound into a structured environment instead of a single device without any relationships whatsoever. The service is part of Azure and supports high scalability and re-usability so called “spatially aware experiences”. With other words duplicating real-time experiences around assets (e.g. machines, processes or other) into a model which knows where exactly the data from sensory devices is coming from. More information can be found here.
Keep in mind that Azure Digital Twin service does not deliver a complete Digital Twin solution. It still requires to connect to sensory devices using for example a IoT Hub. That means you will need to setup a IoT Hub. You also need to think about how you want to visualize and interact with your model. Where are you going to output the results? And how do you want to visualize the structured model of people, places and device. So what does Azure Digital Twin service offer? It offers
Modelling the relationships and interactions between people, places and devices using the spatial intelligence graph
Use of predefined and domain specific object models
Secure scalability and reuse for multiple tenants
Custom functions which can be used for changing incoming data or executing checks against incoming data from sensory devices
Automation of device tasks using advanced Azure compute capabilities.
Using the Azure Digital Twin service
Azure Digital Twin is at the moment of writing in preview. it is even extremely difficult to get a service installed via the Azure portal since Microsoft has limited the number of instances allowed per region. Meaning that you will need to wait till someone removes a Azure Digital Twin service from his tenant before you can add one yourself. Hopefully this will be resolved as quickly as possible with more availability or when Azure Digital Twin goes into general availability.
Microsoft has an extensive amount of documentation about how to implement Azure Digital Twins. There are concepts, references and resources available in the following documentation.
There is an interesting tutorial about monitoring a building with a Azure Digital Twin. This is the Microsoft example i spoke earlier about. The tutorial helps you to configure and deploy a already made solution into your Azure Digital Twin service. The first two steps of the tutorial are the most important one. They are roughly laid out below.
Deploy the Azure Digital Twin service by creating a new resource in your Azure portal.
Create an app registration to access the Azure Digital Twin REST API
Grant the right permissions to the app registration
Download the sample code. It consists of two projects. One is used to configure and provision a spatial intelligence graph. The second one is used to simulate devices and sensory data
Configure and provision a spatial intelligence graph
Define conditions to monitor
Create an user-defined function
Simulate device and sensory data using the second project
Run the simulation data
Run the building service to see if there is a room available based on the current sensory data
The tutorial is thorough and self explaining. Therefor i’m not going to explain each step. You will to follow the following tutorial.
There were some things which require a little bit more attention. Those things i will mention here.
You are building an application which will access the Azure Digital Twin Service through the Azure Digital Twin API. This requires to have an app registration in Azure Active Directory which is given read/write permissions to the Azure Digital Twin service. This requires to have administrative rights in your Azure portal. To get it to work i had to specify an additional platform in the app registration under [Your app registration] > Authentication > Add platform. Add the mobile and desktop applications platform. Make sure you add a redirect URI called http://localhost:8080/. In a later stadium you will be changing the appSettings.json file. That configuration file contains a AARedirectURi defined with the same URI.
Based on some organization settings you will require to have administrative consent. You will need to add delegated permissions for read/write access to Azure Digital Twin. Make sure that the Azure Digital Twin permission appears correctly in the list. If not, then use “Grant admin consent for organization” to give the admin consent.
Keep in mind that the demo uses a simulation sample which simulates sensor data and send that to the IoT Hub which is provisioned by the Azure Digital Twin service. No actual devices and sensors are used in the example.
Azure Digital Twin pricing
The Azure Digital Twin service has no upfront costs or termination fees. You only pay per node and message.
A node is a single component in the spatial intelligence graph. Below the Microsoft example showing a Digital Twin for sensory devices in rooms in a building.
Each Tenant, building, floor, room, device and sensor in this Microsoft demo is a node.
Each API call to the Azure Digital Twin API counts for a message. Each communication sent to a device and sensor are counted as message. And messages send from the Azure Digital Twin to other systems count. You can get discounts when using the service extensively. More information about pricing can be found here.
Azure Digital Twin is a useful tool to build Digital Twin solutions for customers. It is mainly self explanatory when using the different use-cases without going to much into technical detail. Azure Digital Twin service delivers an important part of storing the spatial intelligence graph which is a replication of your real world environment. The tutorial is a great example in how to use the Azure Digital Twin Service in combination with simulated IoT data.
https://globalxr.community/wp-content/uploads/2020/05/DT-Blog-Image-4-Illustration-1024x577-1.jpg5771024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2020-05-16 17:12:412020-05-26 21:22:00An introduction to the Microsoft Azure Digital Twin service
I’m honored to have two talks at one of the largest Microsoft 365 online events in the world called Microsoft 365 Virtual Marathon. An 36 hour event happening on May 27th till May 28th 2020. The event is a joined effort between SharePoint Conference and members of the Office 365 community. And that’s not even all. There are keynotes from key Microsoft employees and from thought leaders and members of the community like Jeff Teper, Bill Baer and Naomi Moneypenny. A truly great online event for everybody with an incredible amount of content. Over 300 speakers will deliver more than 400 sessions during this event. And it is all for free!
My sessions are focusing on solving business problems around training, guidance and remote assistance. Microsoft has two major Mixed Reality applications which are part of the Microsoft Dynamics 365 stack. In a time like this these applications are becoming more and more important for organizations.
Getting your employees ready for business using Microsoft Guides
Microsoft offers a broad range of out-of-the-box solutions for the Mixed Reality market. One of them is creating customized training modules for new workers on the factory floor. Using Dynamics 365 and Microsoft HoloLens we are able to create a specific training which allows new workers to be more quickly in learning their daily job. The session contains explanation of the different functionalities of Microsoft Guides and live demonstration of the application.
Empower your workers using Microsoft Remote Assist
Microsoft offers a broad range of solutions modernizing field services with Mixed Reality for technicians. It empowers them by offering modern tools like Mixed Reality devices, Video calls, Annotations and File Sharing capabilities. These tools allow field service workers to solve complex problems even faster, collaborate together with experts and gives them easy access to work orders. During this session we will show you a global overview of Dynamics 365 Remote Assist using Dynamics, Teams and HoloLens.
Join my online sessions
Hopefully you can join one or both of my sessions and learn more about the world of Mixed Reality, Microsoft HoloLens 2 and Microsoft’s great applications to support many industries and sectors. Sessions will be recorded and available at a later time.
https://globalxr.community/wp-content/uploads/2020/05/HoloLens2_Guides_Motorcycle-1024x683-1.jpg6831024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2020-05-10 16:38:532020-05-26 21:22:01Speaking at Microsoft 365 Virtual Marathon about Remote Assist and Guides
More control over your device its operating system
There are situations when you want to have control over your device in terms of operating system. In our case we have been building applications which have functionality depending on accuracy using WorldAnchors. We noticed that WorldAnchors are stable using the RS1 HoloLens device operating system version in combination with a specific Unity version which is used building the Mixed Reality app. As soon as we moved to a newer version of the operating system, WorldAnchors became less stable hence the application did not work properly.
But the problem with a HoloLens development edition is that it updates automatically to a newer version (currently Microsoft has RS5 released) of the operating system. And you can’t do anything about. Yes, you could block specific update URLs in your network to have the device not being updated by Windows Update. But that is not recommended and gives also other problems.
Production apps require Commercial Suite edition
As many of you do, the HoloLens development edition is often used when a Mixed Reality project is started with the customer. But as soon as you go into production with your app, you will be needing a Commercial Suite edition. This is a recommendation by Microsoft. A Commercial Suite edition only differs in software from a Development edition. The hardware is exactly the same. It gives you a ton of extra software options, warranty and support from Microsoft. You can read here more about the differences.
Depending on your scenario you will need some or all of the steps below to achieve your goal.
Getting your device to the right operating system
You will need to execute this step first if you want to have your devices operating system at a specific version. In our case we want to have it to stay at RS1. The only way to accomplish this is flashing your device. For that you will need the Windows Device Recovery Tool and the image of the specific operating system. You can download the tool here. You can download the HoloLens Anniversary Update recovery package here.
Run the tool on your Windows 10 computer. Startup your HoloLens and connect it via the USB cable to that computer. It will be recognized by the tool.
Select your device. The model and current version of the operating system is shown in the tool. Select “Manual package selection” and browse the RS1 recovery page on disk.
Now select “Install software”. the tool will tell you that the version you are going to install is older. Select yes to install the package.
Keep in mind that flashing the device will remove everything from the HoloLens device and does a factory reset. Meaning it is like if it comes out of the box as when you bought it.
Upgrading to a Commercial Suite edition
If you downgraded your device to RS1 make sure that you do NOT execute the configuration flow of the device. Leave the device as it is for now.
It is possible to upgrade an existing Development edition to a Commercial Suite edition. You will need to create a provisioning package which needs to be added to the device. You also need to buy a upgrade license from Microsoft. They will provide you a XML file which you will need to apply.
There is some Microsoft documentation explaining about the provisioning package process. But there is a catch. I noticed if you follow the documentation it fails to add the provisioning package. I will explain below the best steps to get the package provisioned successfully.
To create a package you will need to use the Windows Configuration Designer Tool. The installation process is described here. If you are running Windows 10 like me (which i presume since you are developing for HoloLens) you can download it from the store.
Run the tool. There are some predefined templates present. Do NOT use the one for HoloLens. That one will cause issues and does not allow to define a lot of custom specifications. Select the “Advanced provisioning” instead.
Enter a name and select a folder. When you create the package it will use the folder to store all necessary files. It needs to be a folder. Selecting for example the root or the desktop as output will not work. Click Next to continue.
To apply the new Commercial Suite license file you got from Microsoft requires to set the EditionUpgrade > UpgradeEditionWithLicense. Browse for the xml file and make sure it is added to the selected customizations at the right column of the tool.
Apply policies to prevent updating
If you also wish to add policies for preventing the update mechanism on the HoloLens device you will need to add the following customizations.
Set the policy Policies > Update > AllowAutoUpdate to “Turn off automatic updates” and the policy Policies > Update > AllowMUUpdateService to “No”. In our case since we are setting the policies for a RS1 version we will not see the policies being reflected in the settings screen later on.
Create the provisioning package
When you have finished all your changes to in the Windows Configuration Designer tool it is time to create the provisioning package. Click on the menu item Export > Provisioning package to create it.
Fill in a name and leave all other settings to their defaults. Press the Next button to continue.
Do not change any of the security settings. The package will fail if you select encryption for the package. Press the Next button to continue.
Finally leave the location as specified and press the Next button the generate the provisioning package.
After the package is generated you will find a set of files generated by the tool. The Upgrade-Policies.ppkg is the actual provisioning package which is needed to apply to the HoloLens device.
Apply the provisioning package
The final step is applying the provisioning package to the HoloLens. There are two ways of adding the package to the device. If you are running a RS5 version you will be able to add them via the settings windows on the device. The second method allows you to add the package before the device is configured.
Important! The downside of the second method is that the device need to flashed or needs a factory reset. Both will remove the upgrade license to Commercial Suite edition. Meaning you have to include the license again in the provisioning package.
In our case we need the second method since that is the only way for a RS1 version.
You will need to do the following:
Make sure the device is not configured yet
Connect the device via the USB cable
Start up the device and wait for the blue square which is used to position the HoloLens correctly on your head
Press the power button (on the back) and the volume button down (at the right and close to the back of the head) together once
Wait for the device to be recognized
Open file explorer and browse to the root of the device
Copy the provisioning package file in the root of the device
Press again the power button and the volume button down together once
Now you will get a message asking “Is this package from a source you trust?”
Select “Yes add it”
If everything went well you will see a message telling you that the provisioning succeeded. Press the Continue button to go into the configuration of the device. Finish it till you get the start menu.
Check if upgrade is executed
You will need to go into the settings window of the device to check if the upgrade to Commercial Suite edition has succeeded. If it contains VPN Connection functionality under network you can be sure it has worked. In case of a RS1 version that will be the only way to check if it worked.
https://globalxr.community/wp-content/uploads/2018/12/recoverytool1-1024x599-1.png5991024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-12-05 20:01:572020-05-26 21:22:38Upgrade your HoloLens development edition to a Commercial Suite edition and keep it on a specific operating system
I’m doing a lot of Mixed Reality presentations. Most of them involve demonstrations with the HoloLens device. All my demos are live using the camera stream of the HoloLens. But as many of you know it has a rather large latency. In some cases more than 5-10 seconds. I tried a lot! Even start using a separate router which i bring with me to every event. But still the latency is bad.
But know it is not anymore!! Currently i’m running on the RS5 preview build on one of our HoloLens devices. And i have to say it is possible to have a live presentation feed of the HoloLens device camera without any latency. Since the RS5 build you have the option to connect to a wireless display or audio device. And that is cool because you can make your laptop also a wireless display.
In Windows 10 there is an option under “System” called “Projecting to this PC”. It has a number of settings which allows you to enable your PC as a wireless display as you can see above.
Now start an app on your HoloLens. Since this build they have created an additional menu which is shown when you use the bloom gesture during a running app. At the right there is a button with a WiFi symbol on it. Select it and a dialog will appear letting you to select a wireless display.
If your PC is configured to be a wireless display and it is on the same network it will appear in the list. As soon as you connect to your PC the video feed is streamed live to your PC without any latency. Depending on the settings made in “Projecting to this PC” a pin is requested and or access is asked before the stream is shown.
Great work Microsoft and the HoloLens team! Finally my demos without any latency!
https://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.png00Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-10-01 17:08:022020-05-26 21:22:38HoloLens and Unity tip #008 – Giving kick-ass Mixed Reality presentations with HoloLens without any latency
I have done a lot with SharePoint in the past. Since the last two years i have been heavily involved in mixed reality projects specifically for HoloLens and immersive headsets. So it comes to no surprise that when a got a glimpse of SharePoint Spaces at the SharePoint Conference 2018 keynote it made me a happy guy. SharePoint Spaces is an immersive and engaging mixed Reality experience for everyone on any device. SharePoint spaces is powered by AI and can be extended by using the SharePoint Framework. It contains the ability of smart templates which leverages your existing SharePoint sites.
I wrote a dutch blog post about how SharePoint Spaces work. You can read it (use Google translate if necessary) here.
Keep in mind that this post is an opinion of my own and not by Microsoft. SharePoint Space is currently in private preview and not available to everyone. 🙂
Finally for me the missing link is filled in. Finally we can work in an immersive way. Finally we are bringing the Office world closer to the worker in the organisation. Why? SharePoint Spaces allows you to have an immersive control over your content. Think of using you content in a three dimensional way which allows you to have control over it in a totally different manner. And on top that, you don’t even need a VR, AR or MR device. You can even use your own desktop because it runs in your browser using WebVR. it is build on an open source framework called babylon.JS.
When i think about SharePoint Spaces all kind of applications and ideas pops up in my head. There are endless possibilities which will empower first line workers and workers on the floor. It also fits in the strategy of Microsoft. The focus with Mixed Reality and HoloLens in particular is in the Modern Workplace. Empowering the workers on the floor using Mixed Reality devices to optimize and improve their business processes. And now we have SharePoint Spaces, the missing link between the Office 365 platform and floor workers. The reason why i mention Office 365 is that it hopefully will extend from SharePoint to Office products like Word, PowerPoint and Visio. The latter i know they are working on.
Lets talk about some possibilities and applications in which SharePoint Spaces can play a role.
First of all think about those applications storing 3D content in Azure for your Mixed Reality apps. With SharePoint Spaces i can actually put 3D content in SharePoint and maintain it. For flat data this is great! You don’t need to build something to maintain the content because SharePoint has it all!
The ability to build 3D SharePoint components by using the SharePoint Framework.
Possible to extend your environment by data coming out of the unified Microsoft Graph. For example people and their relations.
Reuse of 3D model data. i can build different applications in SharePoint, Office and on devices to use the data in some sort way. The cool thing about it that i will reach every target group since it will run in the browser, on a VR immersive device and on AR/MR enabled devices.
Sharing products, designs and other 3D model stuff with your customers. Form discussions around it by using for example Yammer or Microsoft Teams.
Build a catalog for different reasons. Think of using a catalog for a warehouse or store to maintain all product data. Sales and floor workers now have a similar image about the products. Can you imagine order pickers using MR devices to know what next item they have to pick from the shelves because they see a 3D model? That same 3D model, stored in SharePoint Spaces is used by other departments in the office to check their availability, if it needs to be reordered, if there are other same type of components and see first hand differences because they can view it in 3D instead of only working with product numbers and a vague image.
Building apps for design purposes. Design new products and discuss different stages of the model with different roles in your organisation. Make it easier to understand what is 3D printed by seeing the design in front. Think further… what if you start storing different building blocks for modelling a new house and/or the interior.
Consumer apps which allow consumers to experience firsthand their selections. For example an auto car dealer which allows you to design your car and let you walk around it in VR/AR/MR before you buy.
Experience data stored in SharePoint or Excel differently. Instead of flat data it projects the data in a requested 3D graph in front of you and allows you to manipulate the data. Data changes are then automatically brought back into the Excel which is stored in SharePoint.
Workers on the floor can use models from SharePoint Spaces to execute repairs or get an explanations of how it need to be reassembled.
Training of new employees to learn about the products in the company they start working for. This allows us to inform & engage new employees in the organisation.
Following a safety training without using the actual environment. In some cases it is not possible to enter the actual environment. Think of a cleaning room in a hospital or a hazardous location.
Working together on content using any type of device and any type of space. For example one person using a HoloLens, the other a mobile AR enabled phone and the third is sitting behind his desk using his laptop. Working together will not be the same anymore as we are used to.
360 videos allows you to enter whole new worlds through your VR headset. Have virtual tours of buildings or the company you are working for. See the factory floor when you are new to the company. Visit rooms which you are normally not allowed to access due to highly risks of health.
And i can continue on and on. 🙂
Hopefully this post gives you some ideas about SharePoint Spaces and how it will be used in the future. You can read more and learn more about SharePoint Spaces at the following link which is just released at Microsoft Ignite 2018. Some parts can be viewed using an immersive headset.
Below you have the timeline of SharePoint Spaces which is mentioned in one of the sessions at the Microsoft Ignite 2018 event.
https://globalxr.community/wp-content/uploads/2018/09/sharepointspaces-3-1024x496-1.png4961024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-09-25 19:38:342020-05-26 21:22:38A glimpse of the future with SharePoint Spaces
In short Vuforia allows you to recognize 2D or 3D objects inside your application and allows you to map exactly at that position a part of your scene. It uses the camera of your device to do that. It actually locks the camera, since the camera can only be used by one process. In case of HoloLens you are not able to use the camera for other purposes like cognitive services, demonstrations via video streaming, etc. The other issue is that it will use some resources on the background because the process of recognition by Vuforia engine is still active. This can influence the performance of your app.
In older releases of Unity were you had to install Vuforia separately, Vuforia was enabled per scene. Since Vuforia is fully integrated in Unity, it can only be enabled for all the scenes together.
So it is important to turn the behaviour of Vuforia off when it is not needed. There are a numerous of Vuforia classes which allows you to configure Vuforia. And in most cases it is not actually clear how you can turn the behaviour off or on. Some of the methods even cause Vuforia to become unstable.
But turning the behaviour of Vuforia on or off is very simple. If you have Vuforia enabled in your project it configures some default stuff and adds a specific component to your camera. Open the camera in your hierarchy.
You will notice a component added called “Vuforia Behaviour”. That component controls the behaviour of Vuforia when the app is running.
If you want to turn the behaviour on or off you can simply enable or disable the component. Doing this in code is even simpler. There is an instance of the VuforiaBehaviour object available. By setting the property enabled = true or enabled = false will turn the behavior on and off.
https://globalxr.community/wp-content/uploads/2018/09/Vuforia-camera.png198909Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-09-24 09:45:132020-05-26 21:22:38HoloLens and Unity tip #007 – Vuforia behaviour off and on in your HoloLens app
De kracht van Augmented Reality (AR) en Mixed Reality (MR) heeft zich in de afgelopen jaren duidelijk laten gelden. Waar de technologie voor het eerst aan het grote publiek kenbaar werd gemaakt in de vorm van games op de mobiele telefoon, zijn er nu vele toepassingen mogelijk die ook voor de zakelijke markt interessant kunnen zijn.
Wanneer we echter kijken naar de aanschaf van een HoloLens device, dan zijn de kosten aanzienlijk. Zeker als je bedenkt dat er ook nog applicaties moeten worden ontwikkeld. Vaak is het de bedoeling dat meerdere medewerkers gaan werken met de HoloLens, wat inhoudt dat er soms enkele tientallen HoloLenzen nodig zijn om daadwerkelijk impact te kunnen hebben op de werkvloer.
https://globalxr.community/wp-content/uploads/2018/09/hololens-remote-assist-1024x575-1.jpg5751024Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-09-17 08:17:202020-05-26 21:22:38Augmented Reality applicaties voor mobiele devices als eerste stap naar een HoloLens
It is important to understand that not all speech commands on a HoloLens will work without an internet connection. This blog post will go deeper into what needs an internet connection or not.
HoloLens has next to gestures also the possibility to use speech to interact with the device, programs and particular you own app. It is very important to understand what is possible. In principle the device allows you to do the following:
Basic commands – These allows you to get around faster on the device and is used for example to replace the gaze and tap gestures. It also allows you to control the location and size of a hologram. One in particular is really cool and is called “Face me” to let the hologram face you directly. Some examples are:
Button commands – When you hover over buttons, you will notice that it shows you a speech command. This allows you to control that button with speech. It also allows you to start an app on the device
Dictation – This allows you to dictate and translate it to text. Dictation uses explicit punctuation’s and expect from you that you say the name of that punctuation. Think off “Period”, “Comma”, “New line”, “Open Quote” and many others.
App – You can build in your own commands into your app. Coding this is simple, flexible and allows you to almost anything. How to implement is explained here
Many of the commands like the basic commands and button commands can be used without an internet connection. That is resolved internally by the device. It is however very important to use the exact word(s) in the command, otherwise it does not recognize it
But as soon as you start using dictation of specific custom commands in your app you will need an internet connection. The HoloLens uses a Cloud speech recognition service to help to translate the voice command to text.
More information about voice commands can be found here and here
https://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.png00Alexander Meijershttps://globalxr.community/wp-content/uploads/2019/12/FullColor_TransparentBg_1024x1024_72dpi-e1577305903226.pngAlexander Meijers2018-09-07 17:00:302020-05-26 21:22:39HoloLens and Unity tip #006 – Speech commands for HoloLens with and without internet connection