IBM Watson and Unity
After watching the Sprint Planning Meeting video and then reading through the Task Board article, I logged onto my Trello board, looked at through my ‘Explore Ideas’ and identified my next sprint goal – Speech and IBM Watson Unity SDK. The goal is exploring the idea of how player and AI character speech can create a stronger sense of immersion within a VR space.
MI started with a user story centered on a person wants to interact within VR using speech, similar to Alexa or Google Assistant but a bit more robust . User W: “I want to talk with the characters I meet in the virtual world.”
The first step is to setup a Unity scene and get IBM Watson to recognize speech to text and display the text to UI text field. The ultimate goal is triggering in-game logic with a virtual character that could respond to users’ verbal actions. This could lead to a language learning environment where students could learn a vocabulary simply by interacting with an AI character.
Having moved the Trello card to the started board, I set up a free IBM Cloud account to access Watson. I have planned to spend 8 hours within a one week period, on the initial setup. Speech recognition is one of the core features of the app and will give me basic structure to build on. The success of the sprint hinges on communicating with the speech to text server, and it able recognize verbal input through Unity.
I will know I have achieved my sprint goal when the Watson servers are able to transcribe the speech as text and display the output in text field within Unity. The goal is realistic because of the small steps approach I have used in the past. I set have planned to spend 2 two hour blocks during the week and a four hour block of time on the weekend. I will update my Trello board and I am using Unity Collaborate as my version tracking system. Keeping to my time schedule, updating my Trello board and staying focused on simply communicating with Watson, will allow me to successfully complete my sprint goal.
I was able to achieve my sprint goal within the time frame I set because just focused on one basic part of having AI assistant answer questions. I have posted a video below of the finished sprint. Now, that I have the Speech-To-Text that displays in a UI text field, I will start on the next step of getting the Text-To-Speech able to output the Speech-To-Text stream. Below is the working Speech-To-Text module in Unity.
After looking at the list of challenges I the decided on,
- Spend 20 minutes undertaking the ‘I’ phase (inspiration) of the ICEDIP method for a new idea for an app, and then spend 10 mins evaluating the results.
and followed Geoff’s advice to ” Let yourself off the leash!”. I’ve been thinking a lot about “Presence” (spatial immersion for the technically minded) and the users POV, so this activity sounded really interesting. I put the timer on and got started. After twenty minutes I had a paper full of ideas that started off safe but moved into the “what if” realm of thought.
A few of the interesting ones where:
- Having an AI simulation that can interact and respond to your actions or reactions using your avatar or voice.
- Learning a language by being dropped into a “virtual city” where everyone interacts with you in the language you’ve chosen.
- A VR app that can “read” your dreams so you can explore them again.
- An AI / VR app that lets you talk with people from the past – you could have a chat with Jean Baudrillard about reality.
- Having mixed reality pets that have persistence in the world and age.
- A mixed-reality baby sitter for young children.
Spending a few minutes looking back on them, it would be really fun to be able to breath life into these ideas. I liked the idea of being able to sort through your dreams or of even creating a dream scrapbook you could share with people. One or two my be possible in a very limited way but overall it was fun to think about and gave me an idea or two play with. The challenge now would be to come up with a business plans to pitch on of these ideas and research needed to support it.
Having said that, I have started researching the idea of a VR language learning app in which users could converse with an AI character. Having an AI character like Miquela Sousa that students could learn with would be really intriguing experience. So, I started exploring just what speech capable AI’s are available and are able to integrate into the Unity platform. Microsoft has Azure client and Amazon has Sumerian but that service may only run on AWS servers. There IBM Watson and SmartBody by the University of California. At the moment, IBM Watson looks to be the most promising because of their recent partnership, bringing Watson’s AI functionality to Unity’s gaming engine, with built-in VR/AR features. I have added this to my Trello board for now and will continue to follow it.
Player Movement Sprint
The challenge that I have set for the week deals with evaluating user movement. How the user moves through and interacts with the VR environment and how this affects comfort and engagement. I will be gathering feedback from my students
For the sprint, I will use a simple unity environment and duplicate it, then implement teleportation and thumbstick walking in the two respective scenes for use with the new style of Oculus controllers. First, I’ll start by reading Oculus VR best practices documentation and additional resources on the use of movement types and then create the VR test environment. Finally, I’ll explore the two movement types inside VR using an Oculus Rift S.
The sprint goal is to understand how movement affects user engagement with how VR cameras and Asynchronous Spacewarp (Oculus Rift) affect user perception. It also looks at how quickly moving or rotating the horizon line, or large objects affect the player comfort. By importing the Unity Locomotion sample scene as the base environment, and it will be relatively easy to build out the test environment quickly. I have committed 8 hours to the project. I am assigning 4 hours to research and the initial scene buildout. An additional 2 hours to implement the movement control types in the duplicated scenes and finally, committing the final 2 hours to test and recording the results.
The projected 8 hours should be more than enough to accomplish the sprint goal. I have experience with controller mapping inside a unity scene and will not be using additional software or addons. The sprint will conclude after the testing phase, and the next steps are dependent upon reviewing the sprint outcomes.
Link to my Trello card: https://trello.com/c/jVvhFYtf (Links to an external site.)
This week was spent looking at the importance of Persona and why they are necessary and before diving in and developing. Some things for guiding the persona where:
- Who – the person using the app
- Wants – how does this app beneficial to their life?
- Discovery – how would they access this app?
I hadn’t realized how important persona where in the success of an app and how they helped the development team make design decisions based on the user needs. Also, how important those stories were to the sprint process and keeping people focused on the needs of the target audience. I see how this could keep a project on track and help prevent feature creep that could waste time and money.
I didn’t have a personal project in mind when thinking about this assignment, but did have some ideas based on recent events at work dealing with students and SNS problems. Getting started, I put a real name to my persona and what they wanted to accomplish and what motivated them. I looked for a picture that went well with the persona I had outlined and that would provide an emotional pull for the team using it.
The persona is tied to the idea of SNS’s and the app I choose is YouTube Kids (Links to an external site.)and focused on the idea of giving working parents ‘peace of mind’ when comes to their kids net viewing habits. Something that a majority of parents worry about and but unsure about their choices are in providing safe apps for their children to use. A multitude of websites advise parents on responsible tech usage for children and Google’s YouTube Kids helps parents with age-appropriate video. Google wants to “make it safer and simpler for kids to explore the world through online video” and at the same time give parents ” a whole suite of parental controls, so they can tailor the experience to their family’s needs.”
My persona is Lykke Li, a working mother worried about giving her children the freedom to watch YouTube but in a safe and controlled environment.
I also spent time reviewing Unit 8 again because I hadn’t put a lot of thought into gathering user research and the different qualitative methods used to gather users opinions about an app. I going to take Erik Geelhoed’s advice and use interviewing as my main approach to recording and analysing people’s feedback. I teach at high school and the students always have an opinion on just about everything. So, when the time comes for testing my app, I have a ready group of testers. This is something for a future sprint where I set aside some time and write out some simple questions and remember to keep them short and to the point. I will add this sprint idea to my Trello board for the future.
YouTubeKids (persona PDF)
New Goal, Recent Work and Looking back
The unit 6 video with Rich Barham speaking on business plans and pitching really got me thinking about what kind of app I would like to make and how to go about presenting it. As of now, I know I’m looking to make an educational app using VR and at the moment it centered on the idea of creating a collaborative virtual work space where two students could work together fabricating 3D objects that could be saved out to a server and then printed on 3D printers. Inside the space students would be working with apps like, Tilt Brush, Medium and Blocks. After they would be able to save their work to a cloud space like Poly where they could download and print out their work on a local 3D printer or send it to a print service. This is a very rough idea that I have mentioned in our weekly meeting. The challenge going foreword is spent time flushing out the idea and then talking with friends for feedback before making a sprint goal for it.
Also, looking back on my latest sprint, VR movement using the VRTK framework, I able to accomplished two of the three goals (teleportation and object interaction) within the time frame I had set. The third goal of triggering sound effects remains unfinished. I tied sound to actions but was having an issue of syncing the audio to the event action. I will revisit using audio after I’ve had some time to dig a little deeper into how audio and audio effects work in Unity. Later, I can make interactions with audio effects part of a future sprint. Having a smooth integration between visual selection (when an object is selected) and an audio selection response to that interaction, will help reinforce immersion in the environment. The core issue I encountered audio was a lack of understand how audio in Unity really works. While adding ambient sounds to an environment was pretty straight forward, get audio events to trigger at a precise moments, was a bit more involved when I was trying to match animations to event triggers. I had looked at how people had implemented character interaction with game objects in RPG games and thought I could transfer something similar to a VR environment using hand controllers but that didn’t really work.
I am also thinking about how relevant have a deep understand of Unity audio is to my overall goals. This is quite similar to my experience with Maya. It is good to have a general understanding of the basic concepts but there wasn’t any need to be building my own characters when it would be more productive to contract that out and stay focused on my core goals. I’m thinking that contracting out the audio effects would let me spend more time working on the fundamental interactions.
Recently, I have been looking back on I’ve accomplished in the program for my week 6 video reflection. I started off the course with just the basic knowledge of Unity and realized that I would need to spend some serious time learning Unity because I didn’t understand how to use assets correctly or what a prefab was. Also, I was bit clueless on using simple game object actions with scripts.
After our first app jam in week 3, I knew that I didn’t have the depth of knowledge I needed with Unity and some of the important addons its uses( Probuilder, Progrids, Maya integration, and VRTK). I also wasn’t managing my time efficiently while learning and making. Going forward, I am studying Unity using Pluralsight and Lynda. I am focused on just the fundamental training courses and their examples without going off and exploring “too much.” To keep me on track, time management is the key, and I have set up a Trello board, also use a web app called Clockify to track my time. I’ve kept it to two-hour blocks in the evenings that can be increased on the weekend depending on other commitments. By the end of the week, my goal is to have finished a Pluralsight module and have spent time building a working example in Unity. Keeping a daily rhythm, setting deadlines with the Pluralsight modules, and keeping my timeline realistic will keep me motivated over the rest of the course. Learning to use the new concepts I’ve gained in the upcoming weeks has me excited.
I have been on track with my Unity studies using Trello. I am about to start a short sprint goal of testing what I’ve learned with VRTK. I have completed a few of the tutorial videos and have successfully installed the toolkit into an example project. The challenge now is to be able to repeat the process on a new project and then tie user interactions to my own assets rather than those used in the tutorial.
The sprint will have a time frame of one-week, with the total amount of time being 6 hours. This deadline for creating a simple working environment and importing the VRTK framework to the project is quite reasonable. The only issues I foresee at the moment is having to fine tune the VRTK components to get the assets to work correctly with the framework. First, I’ll need to clone VRTK into Unity from their github repository and then setup teleportation markers around the environment and test teleporting to them. After getting movement setup, I begin getting the controllers enabled and testing whether I can interact with objects. Having practiced with the examples, I expect these two goals to easily accomplished within the time frame I have set and should a good challenge. This sprint challenge is a test of what I have learned so far and a signal that it’s time to push myself a bit harder with the Unity. I am keeping the task realistic because this approach works well for me and keeps me focused.
I set have planned to spend 9 hours split into three hour blocks during the week and have some free time on the weekend in case I have to move one of the three hour blocks to the weekend. This will be a one week sprint and I will post the sprint to my Trello board to keep track of my progress. The goal is to be able to build basic interactive environments without having to always reference my training materials which means being able to iterate quicker and start working on projects with more confidence. I believe that the goal is resonalbe and easily achiveable given the amount of time I have set for the project and my recent practice with using the toolkit. Also, the VRTK developers also have a very active Discord giving me an additional resource for help should I need it.
I will update this post with a picture from the finished sprint.
Maya and Character Rigging
Spent the weekend doing some research after getting Maya installed and finishing a few of the tutorials. Its a really amazing what you can do with this and I’ve just scratched the surface. My goal is to get a basic understanding of the building a character and rigging it. I also installed Mudbox and was interested in how you export out the files as .fbx because the files in Mudbox save as .mud. There are some tutorials with Mudbox on the sight but I think I will play with that another day. I am exploring Substance Alchemist which can extract textures form scans which is pretty cool. I need to stay focused and come back to the other Maya workflows later when I more time to dive in.
Working now on building a simple character and its going pretty well, just taking a bit of time with the back and forth of moving through the tutorials.
Update: When I started exploring Maya, I thought I could somehow find the time to learn but after working with it for a week or so realized that it was a lot harder to use than I imagined. I wasn’t able to translate the image I had in my mind’s eye into the character I created with Maya. Then something that I hadn’t realized until my adviser mentioned was that it would be better to contract out work like that, rather than to spend the time that I didn’t have and wasn’t contributing my core goal of making a virtual reality app. In hindsight, it was good to have gained a basic understanding of how Maya worked and general insight into how character movement works. Having this working knowledge provides an opportunity for team communication, but more importantly give me the knowledge needed to communicate the what I am looking for when contracting out Maya work from a freelancer.