Sim Pueblo

Challenges and Methods for Creating a Physical/Digital User Interface

Introduction

Creating a user interface that combines physical and digital components presents a wide range of difficulties, both technical and in terms of creating usable design.  The Sim Pueblo interactive developed for the Coronado Historic Site is an example of an interactive that showcases the challenges and possible solutions to those challenges.

When visitors go to the Coronado Historic Site in Bernalillo, NM, they see the footprint of the Kuaua pueblo, but most of the buildings are no longer there. The staff of the historic site wanted a way to help visitors to envision what it would have looked like 700 years ago.  Its name being a nod to the Maxis SimCity™ games, Sim Pueblo aims to bring this ancient pueblo back to life.  What makes Sim Pueblo unique among many museum kiosk-based interactives is that it contains a physical component in addition to the digital program.  The primary way visitors interact with Sim Pueblo is with a set of 3D printed models of the original structures.  These models were designed by Craig Cassidy and were modeled on the current Taos Pueblo structures.  They allow the visitors to see and feel what the original pueblo could have looked like structurally, but they do not provide much information about the people who lived in them.  The digital part of the interactive adds context and life to the buildings by giving the visitor information about the Kuauans’ day-to-day life.

In order to meet the original goals of the interactive and create the best experience for the users, we selected an emerging hardware platform called an HP Sprout, an all-in-one PC with an upper touch screen, lower touch mat, and downward facing camera.  It is a rather promising platform, but is not without its quirks.  The vast majority of my research was exploring the different ways to differentiate and track the 3D models, and then trying to optimize the user experience for such a unique platform.

Background and the Beginning of Sim Pueblo

All of the Background and Beginning of Sim Pueblo section originally appeared as a paper for Museums and the Web 2016.[1]

A brief history of the Kuaua Village

In 1540, while searching for the Seven Cities of Gold, Spanish explorer Vasquez de Coronado came upon Kuaua Village in what is now northern New Mexico. Coronado Historic Site, although named for the explorer, is today the site of the ruins of this village. At the time of its discovery by Coronado, the village was already generations old (estimated foundation ca. 1325), but it would soon be abandoned forever. Kuaua was one of twelve Tiwa villages in this area. Coronado chose to spend the winter of 1540 here, and tensions soon rose between the indigenous people and the explorer’s party. The stay culminated in the Tiguex War between Coronado’s men and several of the surrounding Tiwa villages near what is now Albuquerque. The lasting effects of the war and subsequent missions are believed to have led to the abandonment of this group of villages less than a century after Coronado’s arrival. The Tiwa people moved elsewhere and still reside in Taos, Picuris, Sandia, and Isleta.

Kuaua_ruins,_Coronado_State_Monument,_1940

Figure 1: Aerial view of Kuaua ruins, circa 1940

In the 1930s, archaeologists Edgar Lee Hewett and Marjorie F. Tichy led an excavation of the site, discovering a series of pre-1492 murals from the pueblo’s southern kiva. Fourteen of these murals have been restored and are displayed in the Coronado visitor center. Prior to these excavations, Hewett was instrumental in crafting the Antiquities Act and served as the first president of New Mexico Normal School (now New Mexico Highlands University).

In 1940, Coronado Historic Site became New Mexico’s first state historic site. Many of the excavated rooms from the village had to be filled in again to preserve their fragile state. A visitor’s center designed by noted Southwest architect John Gaw Meem stands near the buried ruins today, giving visitors a glimpse at the past around them.

Redesigning the visitor center

In the spring of 2015, an exhibition design class from New Mexico Highlands University’s Media Arts & Technology Department redesigned the visitor center. Among other updates, an interactive Sim Pueblo was introduced. In this interactive, visitors can place 3D-printed models of pueblo structures onto the iPad surface, allowing them to build up the village as they wish using pieces of various size. Each piece placed onto the screen will register with the application, and corresponding information about the village through its centuries of growth will be presented to the user. The goal is to allow the user to control the growth of the pueblo and learn about the actual history of its growth at the same time.

The iPad app was a success but did not align to the original vision. In the summer of 2015, Craig Cassidy, the lead designer of the iPad interactive, teamed with Simone Seagle to address the shortcomings and revisit the project with a new, promising technology.

SimPuebCloseUp

Figure 2: Sim Pueblo iPad app in use at the visitor center

iPad limitations

We identified a lot of user-interface constraints on the iPad version of Sim Pueblo. First and foremost were the hardware limitations of the iPad, both of its size and of the way the buildings were originally detected.

The physical size was an issue for the iPad because the buildings, while printed very small, still took up most of the screen, leaving only one small sidebar for the educational content. When visitors were building the pueblo, their arms would cover the small area of the screen where information about the pueblo was being displayed. Often they would miss the text entirely.

One of the biggest limitations was that the iPad could not produce true object recognition. The iPad uses a capacitive touch screen, which means something that can disrupt the electromagnetic field created by the device (such as a finger) can be tracked as it moves across the screen by sensing where the field is being disrupted. The plastic piece alone will not disrupt the field to be sensed by the device. The application team decided to extend the field from the user’s finger to the piece by affixing a copper girdle around the plastic piece, but a building would only register as a touch while a visitor was holding it by this girdle. Therefore, even though the iPad has eleven simultaneous touch points that it can register, the app was only using one touch point throughout the entire cycle of the user placing buildings on the app. This posed a number of problems:

These problems inherent in the hardware made it obvious that the only way to drastically improve the interactive was to move to a different platform entirely.

Goals for Sim Pueblo 2.0

In order to combat the usability issues with the iPad implementation, we came up with a set of goals to meet with the new version:

The remainder of this paper is an exploration of how we strove to meet these goals.

Hardware consideration and selection

In order to really improve the Sim Pueblo interactive over what was available in the iPad version, we had to select a completely different hardware platform. We considered three main options.

First, we considered building a custom solution with a computer, a 3D camera like a Leap or a Kinect, and a touchscreen. This would be very cost effective, somewhere on the order of $1,000 total, but could be difficult to maintain long term because there is minimal support available for the end user. Furthermore, without out-of-the-box hardware, it could be difficult for future developers to recreate the model solution for other historic sites.

The second idea was to buy a touch table. Touch tables allow for a large working area, a built-in computer, support from the manufacturer, and a large number of available touch points. This path was ruled out for two reasons: first, the price was prohibitively expensive (approximately $8,000 for an appropriate table); and second, the vast majority of many-point touch tables are also capacitive touch like the iPad. Therefore, it is not obvious that they would have solved the original problem of only recognizing buildings while they were being held.

The third option, and the solution that we settled on, was to use an HP Sprout. The Sprout is an all-in-one PC that includes a touch screen, downward-facing webcam, downward-facing Intel® RealSense™ 3D camera, and touch mat with projected imagery. This is a reasonably priced (approximately $1,900) out-of-the-box solution that comes with the support and warranty of a name-brand manufacturer in case of hardware issues. It allows for the monitoring of 3D objects on its mat and has a second screen that can be used solely for educational content.

Sprout Experimentation

After we selected the HP Sprout as our preferred hardware, I needed to figure out how to use the Sprout’s native capabilities to track the buildings.  The first step was to decide what language to develop the application in.

API Selection

The SDK (Software Development Kit) has bindings for a small list of languages: a JavaScript API (Application Programming Interface), a CLR (Common Language Runtime) binding for C#/Visual Basic, Unity, and a C++/QML interface.  My first thought was to use the JavaScript API as that language would be easy to pick up for any future developers on the project, it is cross-platform, and there are many free IDEs (integrated development environments) available.  The JavaScript API relies on nw.js, which is an application runtime that uses Node.js to create desktop applications.

However, as of the time of this paper, the Sprout SDK has not yet exposed all of its functionality to the JavaScript API[2]. Furthermore, I could not get the object tracking sample code from their website to work properly on the Sprout.  As that code is the most important functionality I needed to program Sim Pueblo, I was forced to rule out JavaScript as an option.

Around the same time I was investigating the different APIs that were available (Fall 2015), Microsoft released Visual Studio Community 2015, the newest free version of their Visual Studio product.  Community 2015 is the first free version of Visual Studio that contains software called Blend.  Blend was originally sold as a stand-alone (very expensive) product used for doing design in WPF (Windows Presentation Foundation) and Silverlight projects.  Later, when Microsoft began packaging it with the paid versions of Visual Studio, but now it is freely available.  It allows Adobe Flash-like animation and design capability that I was already quite familiar with and I knew would be very useful for creating a compelling UI for Sim Pueblo.

Between the failure of the JavaScript sample code and the release of Community 2015, I selected the CLR binding and chose to produce the application in WPF.   WPF is a Windows desktop only development framework with very strong UI capabilities and powerful back-end programming functionality, but it is extremely limited by its possible platforms - only Windows machines.  For this project it would be perfect, because the project is designed to run exclusively on an HP Sprout, all Sprouts run on Windows, and all of the Sprout’s functionality is available for that binding.[3]

In WPF, all design elements are defined in a special dialect of XML called XAML, or eXtensible Application Markup Language.  This includes all UI components like grids, drawings, and buttons as well as storyboards, visual states, behaviors, and styles.  All of the code-behind is written in either C# or Visual Basic, but most developers use C#. This allows for powerful design and development capabilities.

Object Tracking Attempts and Solution

In order to aid in the technical description that follows, below is a diagram of the Sprout with parts labeled.

Figure 3: An HP Sprout installed at the Coronado Historic Site.

Once the API was selected, it was time to try and get the Sprout to distinguish and track the buildings.  One of the stated capabilities of the Sprout is that it can track physical objects.  I thought that because the buildings were one orientation only (rotation was only in the x-y plane), that it would have no trouble tracking them.  I attempted to train the Sprout to recognize the buildings themselves, but because of parallax when buildings were moved from one side of the mat to the other and changing lighting conditions, it was unable to distinguish them.  Sprout object tracking works by scanning the downward facing webcam’s video frames for a specific bitmap image.  Because of this, the Sprout’s object tracking capability is limited to 2D objects like postcards or photos.  

The Sprout is capable of scanning 3D objects and taking note of their shape, outline, and texture.  The “capture” process (as it is called in the documentation) takes about three to five seconds per scan and whites out the mat screen in the process.  In order to use this, we would have to add a button to tell people to update the mat every time they added a building, which would make for a very disjointed user experience.  Furthermore, the whiting out of the screen is very jarring and could be an accessibility issue for people with epilepsy.  This initial experimentation had disappointing results, so I needed to explore other ways to identify and track the buildings.  I had three unsuccessful attempts at object tracking before settling on my final successful method of using building tags.

Unsuccessful Methods

Outline Method

Each of the pueblo buildings has a different physical outline, so I figured I could use that unique footprint to identify and track them.  In order to do that, I would need to reliably and quickly separate the buildings from the mat background.  The Sprout has a downward-facing webcam and downward-facing Intel® RealSense™ 3D camera that continuously monitors the touch mat. The most obvious thing to do would be to use the 3D camera because no matter what extra content was projected on the mat, the buildings would still stand out.   The RealSense™ camera is similar to a Microsoft Kinect in that it uses an array of infrared laser points that reflect back to a detector and create a point cloud representing the 3D shape of objects within its field of view.

The RealSense™ camera has its own SDK[4], but at the time I was working on this particular aspect of the problem, the Sprout’s software did not allow developers to access any of the RealSense SDK’s capabilities[5].  However, the output of the RealSense camera was still available in its raw form.  I attempted to use that output to isolate the buildings using the series of image filters from the AForge.NET[6] image processing library shown in the following code snippet.  This library was so fast that the operations produced virtually no lag whatsoever, so the frame-by-frame analysis is done in real time. The result of the series of filters is shown after the code sample.

// First use the AForge library to get the frame to be a binary white building on black background image.  The processingRect is the region of the video field that contains the touch mat.

IFilter crop = new Crop(processingRect);

frame = crop.Apply(frame);

           

// First we narrow the image down to just the mid-range green channel because all of the buildings are at about the same height and therefore about the same level for the depth camera.  I sampled the value for the green and got 136 for everything I cared about.

ColorFiltering channelFilter = new ColorFiltering();

channelFilter.Green = new AForge.IntRange(130, 140);

channelFilter.Red = new AForge.IntRange(0, 0);

channelFilter.Blue = new AForge.IntRange(0, 0);

frame = channelFilter.Apply(frame);

           

           

// Convert it to grayscale and add a threshold filter that takes all pixels above a certain value and makes them white, and all below black

Grayscale k = new Grayscale(0.2125, 0.7154, 0.0721);

frame = k.Apply(frame);

IFilter threshBinary = new Threshold(90);

frame = threshBinary.Apply(frame);

           

// Now use an erosion filter to get rid of the diffraction ridges

// This erosion matrix tells the filter to only use the pixels to the top and bottom of the specified pixel.

Int16[,] matrix = new Int16[,] { { 1, 1, 1 }, { 0, 0, 0 }, { 1, 1, 1 } };

Erosion erosion = new Erosion(matrix);

frame = erosion.Apply(frame);

// Now fill in the holes that are smaller than 15 pixels across

FillHoles fillHoles = new FillHoles();

fillHoles.MaxHoleHeight = 15;

fillHoles.MaxHoleWidth = 15;

frame = fillHoles.Apply(frame);

filterStep5_erosion.png

Figure 4: Intel® RealSense™ camera output on left, result of video filters on right

The problem here was that, while the 3D camera does see the buildings, the outlines were subject to electromagnetic effects that did not give them crisp edges, and the smallest building was generally not picked up at all.  Therefore, I decided it would be better to analyze the output from the video footage.

In order to see the buildings clearly in the video feed, I spray painted them a dark brown color so they would contrast with the adobe color projected on the mat.  Using the same image processing library, I took the output of the downward facing webcam and then filtered it to isolate the buildings.  The series of filters that I used for this image is quite similar to the above example, so that code has been omitted.  Once the buildings were isolated from the background, I used a blob detection algorithm to find each building and a Moravec Corner Detection algorithm to get the corners of each one.  With an array of points representing the outlines, the challenge was to sort them into a proper polygon and then figure out a way to differentiate the polygons and associate the different buildings.  Luckily each building’s footprint had a different size, so I was able to guess the buildings fairly reliably using their area.  The code below shows the blob and corner detection function usage (both contained in the AForge.NET library), and the code used to determine area.

MoravecCornersDetector mcd = new MoravecCornersDetector();

mcd.Threshold = 5500;

               

BlobCounter bc = new BlobCounter() {MinWidth = 5, MinHeight = 5, MaxHeight = 110, MaxWidth = 150, FilterBlobs = true};

bc.ProcessImage(frame);

Blob[] blobs = bc.GetObjects(frame, true);

List<int> guesses = new List<int>();

for (int i = 0; i < bc.ObjectsCount; i++)

{

        var blobCorners = mcd.ProcessImage(blobs[i].Image);

        int guess = BuildingUtilities.GuessBuildingByArea(blobCorners, 100);

        guesses.Add(guess);

}

// Guesses which building a blob could be by comparing the areas of the polygon to the area of the archetypal samples to within a given threshold.

public static int GuessBuildingByArea(List<AForge.IntPoint> contour, double confidenceMargin)

{

    int guess = -1;

    // The corners of the polygon are sorted by angle around the center point of the blob.  Unfortunately this is only reliable for convex polygons, not concave ones.

    sortPoints(contour);

    double testArea = calculateArea(contour);

    double areaDelta = 100000000.0;

    for (int i = 0; i < _buildingDatabase.Length; i++)

    {

        double compDelta = Math.Abs( _buildingDatabase[i].Area - testArea);

        if (compDelta < areaDelta)

        {

             areaDelta = compDelta;

             if (compDelta < confidenceMargin)

             // Using the same counting convention as I used when I originally numbered the buildings.  It’s one-based instead of zero-based.

             guess = i +1;

         }

     }

     return guess;

}

The result of the filtered video frame, the blob and corner detection, and the area estimates of the buildings is shown in Figure 5.

lee.fig6_.png

Figure 5: Unfiltered downward facing webcam footage on left, filtered and corner detected image on right.  Numbers represent algorithm guess at building ID based on area of the polygon.

The guesses for the buildings are quite accurate, but from that information we cannot glean information about the building’s rotation. It is also incapable of distinguishing buildings whose footprints are the same area.  In order to improve the detection, I was going to use a Turning Function[7], which is an algorithm for differentiating polygons that is size and rotation invariant.

Even though this is an ideal method for differentiating the buildings because of the unique shapes (they would not have to be adorned in any way), the algorithm would have to become much more complex in order to work whenever the buildings would touch.  Not only can the camera no longer detect the boundary between two buildings, but it also makes it much harder to sort the points of the polygon’s corners properly.  Originally I used a simple algorithm where I sorted the points by angle around a central point, but this does not work for L-shaped buildings because you get results like the ones shown below in Figure 6.

lee.fig7_.png

Figure 6: The same algorithms, but with buildings touching.  The corners of the polygons have become much more difficult to sort properly and the blobs representing the buildings on the left cannot be separated.

Even though the building outlines were unique, once they touched the ability to differentiate them was almost non-existent.  Clearly this method would not be reliable enough to use in the program.

Fiducial Dot Method

The next option was to put some sort of identifying marker on the buildings.  The simplest polygons that can be reliably distinguished from each other are triangles, therefore I put a triangle of dots in a contrasting color on the buildings and then attempted to use the webcam to detect the triangles.  

Using a similar series of filters to the above methods, I isolated the buildings from the background and then tried to locate the specific green of the dots on the buildings.  The AForge library was not able to isolate the color of the contrasting dots very well due to changing light conditions and the webcam’s auto-focus/brightness settings.  That result is shown in Figure 7.

lee.fig9_.png

Figure 7: Raw video frame on left, filtered on right.  The orange rectangles are the program’s guess at dot locations, turquoise lines connect dots for all possible triangles, and numbers represent a guess at building identities based on the geometry of a particular triangle.

While some of the green fiducial dots were picked up by the camera, there were a lot of false positives and missed dots.  The guesses of the different triangles shown above are for the most part very inaccurate.  While the blobs of the buildings were easy to detect, the green dots were too small to work properly.  It also caused problems with processing speed when many dots were detected simultaneously.  Furthermore, the neon green dots on the buildings were distracting, unattractive, and not at all representative of an historical pueblo building.  Due to these problems, I abandoned this line of research in search of another method.

Infra-red Tag Method

A short lived experiment for tracking tags involved getting an IR-reflective tag and using that to fake out the RealSense camera.  The idea was that because the RealSense camera uses IR laser to detect objects, if you had a special IR-reflective tag you would be able to catch that as an anomaly with the camera.  I had hoped that I could do something similar to the fiducial dots but that would be simultaneously invisible to any users and easier to separate from the background.  I ordered a special tag used by the armed forces to determine friendly forces from a tactical weapons store in order to try this out.  Unfortunately, it looked identical to the buildings from the point of view of the camera and was not detected at all.  

The Tag Method and Sprout Object Tracking Library

Finally I decided to go back to using the Sprout’s built-in capabilities to identify and track the buildings.  We found a video produced by Edddison 3D that shows tracking of a 3D object on the Sprout mat[8].  The 3D object they were tracking looked like a cylinder with a unique black and white shape on the top.  I guessed that the Sprout was tracking that unique 2D image, and perhaps I could use the same idea and add an identifying “tag” to each building.  This turned out to be slightly easier said than done, because at the same time I thought of this, I began having problems with the Sprout’s hardware and its SDK.  Those will be outlined in the Hardware Issues section of this paper.  After all of the Sprout’s problems were sorted out, I was able to get this tag method to work.

 There is a little bit of information on what images constitute good objects for tracking in the Sprout Developer Guide’s section on native object tracking:[9]

Sprout recognizes and tracks features in video frames, based on the same features in training images. Features can be thought of as changes in the lightness and darkness of adjacent areas. A photograph of a beige wall has few features. A photograph of orchids and a toucan in a rain forest has more features. Objects with more features are recognized and tracked more reliably than objects with few features.

Edges of a plain object are not good features[.]

A pattern cut from a plain piece of paper has edges with a specific shape, but those edges do not make good features. Drawing lines near the edges increases the precision of recognition and tracking, though even in this case, the number of features might not be enough for object recognition and tracking.

During object recognition and tracking, Sprout does not consider color.

Based on my tag tracking experiments and this text, I found that the best tags are ones with a defined outline and a fair amount of detail in the center part of the image.  The most reliable way that I found of producing the tags was to hand-draw them with a marker, scan my drawings, trace them using Adobe Illustrator, and resize them to fit each building.  A table showing effective and ineffective tags is shown in Figure 8, and Figure 9 shows a sample of the 3D printed buildings with tags on them currently in use at Coronado.

Effective Tags

building6.bmp

15.51.02.27_Object_9.bmp

building7.bmp

15.51.02.26_Object_6.bmp

Ineffective Tags

building2.png

17.05.53.08_Object_1.bmp

17.05.53.08_Object_2.bmp

Figure 8: A table showing sample tags.  Effective tags for the Sprout tend to have substantial outlines and complex components with minimal adornment outside the primary outline.  

buildingsWithTags.jpg

Figure 9: 3D printed buildings with paper tags glued on them.  I used matte finish Mod Podge as an adhesive and sealer because gloss tends to make the tags harder to see for the camera.

I used the Sprout SDK sample code[10] to “train” the Sprout to recognize those specific 2D images.  The sample code takes a bitmap image of each individual object on the mat and then I save them as resources within the project file.  When the time comes to track those images, they are loaded into memory and passed to an instance of a Sprout Object Tracker that locates each image and gives its location in millimeters in three dimensions and its rotation in radians.  The conversion between millimeters and pixels for the mat is a trivial conversion as the exact millimeter size of the Sprout’s touch mat is given in the Developer Guide.[11] 

I organized all of the object tracking code into a separate Class Library project called TagTracker inside the overall solution so that it could be repurposed for future projects.  The tags tracked by the Sprout’s software are stored in a TrackedTag object that keeps track of the object’s previous location, rotation, ID, etc.  When a new tag is observed or a current one has moved, then the Tag Tracker fires a custom event that the main window of Sim Pueblo is listening to.  Below is code that fires whenever the Sprout’s object tracker updates.  It can be found in Sprout_TagObserver.cs of the attached code.

private void TrackingHandler_TrackUpdated(object sender, PcTrackEventArgs e)

{

    if (DateTime.Now.Subtract(lastMessageTime).Milliseconds > 500) // prevent it from firing too often

    {

        lastMessageTime = DateTime.Now;

        if (e.TrackedObjects != null)

        {

            foreach (var trackedObj in e.TrackedObjects)

            {

               

                double centerX = trackedObj.PhysicalBoundaries.Location.X;

                double centerY = trackedObj.PhysicalBoundaries.Location.Y;

                // Try and find the tag by its name which was originally defined by the filename of the tag

                TrackedTag tag = trackedTags.FirstOrDefault(t => t.Name.Equals(trackedObj.Name));

                // We're going to transform the coordinates (which are in mm) to be a range from 0 - 1 in X and Y

                Point tagCenter = new Point(centerX / MatSizeInMM.Width, centerY / MatSizeInMM.Height);

                       

                if (tag != null)

                {

                    // if we have that tag already, update its coordinates and check if it moved by comparing the new value to the last saved one

                    bool moved = tag.UpdateCoordinates(tagCenter, trackedObj.Rotation);

                    if (moved)

                    {

                        // Fire custom event in order to inform Sim Pueblo

                        OnBuildingMoved(new BuildingTrackerEventArgs(tag));

                    }  

                }

                else

                {

                    // Found new tag.  Add it to the list of current tags

                    TrackedTag newTag = new TrackedTag(trackedObj.Name, tagCenter, trackedObj.Rotation);

                    trackedTags.Add(newTag);

                    // Fire custom event in order to inform the Sim Pueblo window

                    OnBuildingAdded(new BuildingTrackerEventArgs(newTag));

                }

            }

        }

    }

}

This method gives a much simpler and more accurate location for each of the buildings than any of the previous methods.  With that information, I am able to track the buildings on a diagram on the upper screen and check the buildings for accuracy on the mat screen.

The tag method appears to have some drawbacks, however.  The documentation does not give a specific limit on how many 2D objects can be tracked at once, but I have noticed with my tags that it does not seem to do a very good job with any more than seven at once.   This leads to some frustrating user interactions as there are nine buildings total.  I have been told by Sprout engineers that the tracking capability has been tried on up to 22 objects successfully, so it could be a problem with the size of the tags or maybe the amount of other visual clutter on the mat.

Another drawback is that I cannot reliably tell if a building has been removed or not.  I attempted to ping each building object saved in code each time it was detected by the object tracker and then remove buildings after a certain amount of time of inactivity, but that was not reliable because it would often not fire a tracking event if the tag hadn’t moved.  Therefore it would remove tags that hadn’t been physically removed from the mat.  I decided that that was a worse user interface flaw than having a removed building still appear.

Hardware Issues

I had a number of issues with the Sprout hardware and software over the course of the research.  When I first got the Sprout, the touch mat would not connect properly.  It would flash on the screen and repeatedly disconnect and then reconnect itself.  I called HP’s customer support and they were very helpful and sent a new touch mat quickly.

In the spring, after having come to the conclusion that the best way to deal with the buildings was to use the Sprout SDK rather than doing my own image processing, the Sprout’s sample code no longer worked.  I got an exception whenever I would run the code stating that the version of the platform referenced by the library was incorrect, but I could not update the platform or the SDK.  When I tried to update the SDK, it would break the Sprout Workspace, meaning I was no longer able to use any of the Sprout’s factory capabilities.  I upgraded to Windows 10 in the hopes it would solve the issues, but that did not work.  After that failed, I tried resetting the Sprout to its factory settings, but the problem persisted.  At that point, I was able to get in contact with representatives from HP who sent a new Sprout since it was still under warranty.  I tried to install the new version of the SDK on the new Sprout and still had no success.  Eventually I was put in contact with a project manager for the SDK who had an engineer remotely log in to the Sprout and reset it.  From that point on, the Sprout worked, but the engineer used some sort of proprietary tool that they uninstalled, and I have no idea what he did, so I don’t know exactly what the problem was.

One recurring problem with the HP Sprout is that Windows Updates can cause the Sprout software to malfunction.  I was warned about this by a representative from HP, but Windows 10 makes updates mandatory.  Any malfunctions are usually fixed by upcoming updates from HP, but they often take a very long time to install and the Coronado Site does not have reliable internet or even power overnight.  This means that the Windows 10 updates usually should not begin, but if they do for some reason, it will be difficult to update the Sprout Workspace and Platform.  The Workspace is currently out of date, but needs about one gigabyte of software updates to get to the most recent version.  The old version does not allow for interaction with the Intel RealSense SDK, so this could be a problem in the future depending on any future changes to the project.

While it was immensely frustrating to be dealing with this while trying to produce the interactive, it reinforced my choice of platforms.  HP’s customer support is very helpful and they were very happy to be involved with the project.  So even though I had hardware problems, I was in contact with a company with the resources and desire to fix them.  

User Interface

All this hard-earned ability to track and identify individual buildings is useless without a good user interface.  The design of the user interface is centered around the idea that the visitor is recreating the experience of living in the Kuaua Pueblo in the past and bringing that ancient village back to life.  

The primary advantage of using a dual-screen platform like the Sprout is that the whole upper screen can be used for educational content and the bottom mat can be used for the buildings and animations that bring the Pueblo to life.  The upper screen displays the text that was originally written for the first Sim Pueblo, the current layout of the buildings, all of the items that can be added to the mat, and instructions.  This way visitors have a full area to work with when putting the pueblo together and have much more screen real estate in order to explore the informational text.

Because of the ubiquity of phones and tablets, most museums visitors are not intimidated by a touch screen interactive, but a dual-screen PC with a camera jutting out is not something that most people have seen.  During user testing, I noticed that while groups of adults are sometimes hesitant to play, younger audiences seem to be intrigued and are eager to try it out.  In order to entice visitors to try it even if they are intimidated, the attract screen is an animated version of the instructions in the hopes that it will give people enough of a hint to begin using it.

Upper Screen

SimPueblo_upperScreen_newAdditions_v2.png

Figure 10: Upper Mat Screen showing five out of nine buildings

Unfortunately, because the original Sim Pueblo added text sequentially, the text that we have does not correlate with individual buildings. Some of the text is chronological, such as information about the original settlement of and departure from the pueblo, but most snippets deal with a facet of pueblo life such as ceremony, architecture, or natural resources.  It was not possible to rewrite it in order to match each individual building because the text has to be approved by tribal elders.  The idea that we settled upon was to rebuild a mural rather than presenting the information sequentially.  This is both functional - because the original text is not chronological, and metaphorical, because it represents the idea of building a picture of life in ancient Kuaua.  The mural image selected by the Coronado staff is a watercolor painting called Corn Grinding by Velino Shije Herrera. 

In order to know when a new piece of the mural should be added or any other part of the UI should be updated, the MainWindow listens to the TagTracker’s custom events.  Whenever the Tag Tracker detects a new building or determines that one has moved, it fires an event that contains information about which building moved and where it went.  The MainWindow’s response is shown in the sample code below from MainWindow.xaml.cs.

private void BuildingTracker_BuildingAdded(object sender, EventArgs e)

{

    // BuildingTrackerEventArgs contains information about that building such as ID, location, rotation, etc

    BuildingTrackerEventArgs bte = e as BuildingTrackerEventArgs;

    Console.WriteLine("We found building number " + bte.ID);

    // If in attract mode, then restart

    if (onAttract)

    {

        // All of the tracking is happening off of the UI thread in for good user experience.  Dispatcher.Invoke() tells WPF to start a given task on the UI thread when it is not busy

        Dispatcher.Invoke(new Action(() => restart()));

        return;

     }

     // This counts as user action, so reset the timeout timer

     resetTimeoutTimer();

           

     Dispatcher.Invoke(new Action(() =>

     {

         // Save our new information

         BuildingVisualizer updatedVis = BuildingLocations[bte.ID];

         updatedVis.Location = bte.Location;

         // The rotation is reflected because the Sprout calculates the rotation from the other side.

         //  The RotationOffset is the rotation of the tag image with respect to the rotation that we should have for the building.  MathsAndPoints is a library I wrote with various useful math equations in it, such as the distance formula and conversion between degrees and radians

         updatedVis.Rotation = 360 - (MathsAndPoints.RadiansToDegrees(bte.Rotation) + updatedVis.RotationOffset);

               

         updatedVis.OnScreen = true;

         // Send information to our map

         puebloMap.UpdateBuildingLocation(updatedVis);

         // Add people to the Mat Window

         MatWindow.AddSomePeople(bte.Location);

         // Add a shard to the MainWindow

         updatedVis.ShardID = addShard();

               

    }));

}

Each time a new building is placed on the mat screen, a new shard pops to the forefront.  It begins by displaying its “back,” which has a snippet of text on it and an explanatory picture.  The user is also notified if their shard has unlocked an additive for the pueblo such as crops or animals. When the shard is closed, it snaps into place, gradually completing the mural.

From a programming perspective, each piece of the mural is an object that inherits from a base object called a Shard.  It has two visual states - one that shows the text, and the other that shows the section of the mural that the shard represents.  The base class takes care of all shared behavior - so for example, bringing shards to the front, sending them to the back, animating them into their proper location, and so on.  It also contains information about whether that shard unlocks items that can be added to the mat window.

In the upper right corner of the screen is a map that shows the current layout of the buildings on the mat.  The main window retains a reference to all of the buildings along with their centers and rotations in terms of percentage from top to bottom and left to right, that way it can pass the coordinates along to both the Pueblo Mat on the upper screen and the Mat Screen for their own transforms.  For example, a building in the center of the mat would be located at (0.5, 0.5) rather than (612, 384), which would be its pixel coordinates on the Mat Screen.

An example of how those coordinates are used can be found in YourPuebloMap.xaml.cs.  For tracking on the mat and comparing against the known building locations, the code is very similar.

public void UpdateBuildingLocation(BuildingVisualizer bv)

{

    // This is the path object that will be moved around the map.

    Path toUpdate = buildingOutlinePaths[bv.ID - 1].Path;

    // The building visualizer is a class that holds the location and rotation of the building as well as whether it is still on the mat

    if (bv.OnScreen == false) toUpdate.Visibility = Visibility.Collapsed;

    else toUpdate.Visibility = Visibility.Visible;

    double scaledX = bv.Location.X * 363.0; // Width of the map in pixels

    double scaledY = bv.Location.Y * 265.0; // Height of the map in pixels

    // I set the render transform origin of the path to match the location of the tag on the building.  This way when the building rotates it will be correct on the screen.

    Point currPoint = new Point(Canvas.GetLeft(toUpdate), Canvas.GetTop(toUpdate));

    Point newPoint = new Point(scaledX - toUpdate.RenderTransformOrigin.X * toUpdate.ActualWidth, scaledY - toUpdate.ActualHeight * toUpdate.RenderTransformOrigin.Y);

    // Animate it if the new point is far away so that it does not appear choppy

    if (MathsAndPoints.Distance(currPoint, newPoint) > 15)

    {

        AnimateToPoint(toUpdate, newPoint, bv.Rotation);

    }

    else

    {

        Canvas.SetLeft(toUpdate, newPoint.X);

        Canvas.SetTop(toUpdate, newPoint.Y);

        buildingOutlinePaths[bv.ID - 1].Rotation.Angle = bv.Rotation;

    }

           

    int visibleCount = 0;

    foreach(BuildingPathInfo path in buildingOutlinePaths)

    {

        if (path.Path.Visibility == Visibility.Visible) visibleCount++;

    }

    // Keep track of how many buildings we have and then display that information.

    BuildingCount = visibleCount;

}

All of the available additives for the pueblo are shown in a menu on the right hand side of the screen.  These additives are the items that people are able to add to the mat in order to bring it to life, such as dogs, crops, fires, pottery, and so on.  They are represented by an array of checkboxes.  When one button is checked, the selected item is sent to the mat window.

This part of the UI has proven to be a bit challenging because visitors tend to miss it.  During the first incarnation of Sim Pueblo, the buttons were only visible when the shard they were associated with was open.  In this state, generally no one would realize they were there.  The next iteration had them appear sequentially as they were unlocked.  In the final update, I am changing it so that they are always visible, but shown as locked until the appropriate number of buildings are added.

I added a function to the UI that is only for the Coronado Historic Site staff.  They requested a way to make the computer easier to shut down, so I programmed a special window with a number pad that pops up when an invisible hit point in the upper right corner is tapped.  When the staff types in their special four-digit code, Sim Pueblo launches a batch script that shuts down the computer.  If the wrong code is entered, nothing will happen.  I designed the interaction so that it would not give visitors any indication of what they should enter or what would happen if they entered the correct number on the off chance that someone opened the window by accident.

Mat Screen

When a visitor approaches Sim Pueblo, the mat is blank other than the outlines of where the buildings are supposed to be placed.  When people pick up the buildings and place them on the mat correctly, the outlines begin to glow yellow to indicate that a building has been placed in its proper position.  Each time a building is added, either two or three people appear from the center of the building and begin wandering around the pueblo.  There is rudimentary artificial intelligence where they decide to either walk around the pueblo randomly or visit the water to gather fish.  When the crops are added, people can go to the crop locations and harvest crops as well.

MatViewV3.png

Figure 11: A sample frame of the mat window in play. Seven out of nine buildings are active.

When users click on an additive on the upper screen, what they selected (animals, sounds objects, etc) is added to the mat screen.  All of the objects on the mat, from the people to the fire pits, inherit from a base class called PuebloCharacter that handles all movement, animation, or lack thereof.  A sample of the code that adds characters to the mat, used for the animals in particular, is shown below.  The code for objects and crops is similar.  

private void AddSomeAnimals(PuebloAdditions.Additives species, int number)

{

    for (int i = 0; i < number; i++)

    {

        // All of the additives inherit from a PuebloCharacter class.

        PuebloCharacter animal = null;

        switch (species)

        {

            case PuebloAdditions.Additives.Dogs:

                animal = new Dog();

                break;

           

            // Turkeys and Rabbits are the same object with the same behavior but with a different appearance

            case PuebloAdditions.Additives.Rabbits:

                animal = new Bunkey(Bunkey.Animals.Rabbit);

                break;

            case PuebloAdditions.Additives.Turkeys:

                animal = new Bunkey(Bunkey.Animals.Turkey);

                break;

            default:

                break;

         }

         if (animal != null)

         {

             // Randomly locate the animal on screen in a location outside the main pueblo

             animal.SetRandomObjectLocation(false);

             // The goal point of the animal is the random location where they are headed

             animal.GoalPoint = animal.GetNewGoalPoint();

             additiveCanvas.Children.Add(animal);

        }

    }

}

User Testing and Feedback

After the interactive was installed for a soft opening on May 31, 2016, I continued to do additional bug testing and usability enhancements.  The first remedial evaluation was performed at Coronado Monument on June 17.  Unfortunately, due to the incredibly hot summer weather, I was only able to interview one set of visitors.  Luckily, that did give me considerable time to talk to docents and staff about ways to improve the program.  Some of the most interesting advice I received was from a man who was selling jewelry at the site.  He was a member of the Santo Domingo Pueblo and pointed out that the original drum sound track was not traditional Native American drumming. He also suggested that I add a celebratory Corn Dance at the end of the interactive, but the staff vetoed this idea.

I also spent considerable time talking to a docent who was a retired early childhood educator who had a lot of insights.  She pointed out that the text was far too long and that the vast majority of visitors had not read it.  From the feedback I received, I modified the text on the shards to have more paragraph breaks and emphasized text even though I could not modify the actual content.

One of the things that I noticed from the few people whom I watched use Sim Pueblo was that the tags did not always track as people expected, so they tended to think that they were not putting the buildings in the correct places. They did not realize that it was the software that was not functioning properly, since the Sprout seems to be limited to tracking about six or seven of the tags properly.  This also tended to frustrate visitors because they did not realize that they were able to get to an end point.

A second evaluation with a new and more permanent set of buildings took place on July 3rd.  Unfortunately, much of the time I had available to do user testing was spent fixing an issue with the Sprout Workspace.  A recent Windows Update broke the version of the Sprout Platform and Workspace so it had stopped functioning properly.  I did not have the time to update the Sprout software to work with the new Windows Update, so instead I reset the Sprout to a restore point from before the Windows Update that caused it to malfunction.

After that work, I was able to spend some time interviewing visitors.  What I noticed from watching people use Sim Pueblo, and also from what I have been told by the staff, is that people tend to spend a long time on it.  The interactive has a very low “bounce rate,” - meaning people that try it, lose interest, and walk away.  All of the interactions with it last at least three minutes, and sometimes as much as twenty.  People seem to be very engaged with it and take their time moving buildings around.  The vast majority of visitors tend to understand all of the main components of the game - placing the buildings, building the mural, etc.  Furthermore, most people seem to understand the point of the game and read at least some of the accompanying text.  

One weak point in the UI design is the Pueblo additions.  One challenge of working with two screens is that people do not always know what all parts of the screen they need to pay attention to.  Many visitors miss the panel on the right where all of the buttons are and do not seem to notice if any new buttons pop up.  Ethan suggested that I always have the additions visible, but show them as locked until the correct number of buildings is added on the mat.  I am addressing this issue in the next update.

Overall, the user testing data (found in the appendix) supports the idea that this interactive successfully engages visitors and teaches them about life in the ancient Kuaua Pueblo.  

Future Considerations

Building Identification and Tracking: An Alternative Approach

As of spring 2016, the ability to use the Intel® RealSense™ camera SDK has become available to Sprout Developers.  The RealSense™ SDK comes with tools that allow for 3D object recognition and tracking.   I used the SDK tools to create a 3D map of all of the pueblo buildings and then saved them.  After that, I loaded the maps in one of the sample tracking programs that came with the SDK.  The object tracker shows the orientation and location of all the building accurately, but unfortunately cannot handle all nine at once.  A screenshot of this program in action is shown in Figure 13.

RealsenseObjectTracking_5or6limit.png

Figure 13: The Intel® RealSense™ SDK’s Object Tracker successfully tracking five out of nine of the pueblo buildings

This would be an ideal solution because it would be fast, reliable and would not require any particular paint scheme or identifying sticker for each building.  However, it would require more research and time to implement.  This sample program shown above is written in C++, but there is also a version written in C# that I could use for Sim Pueblo.  With experimentation, it may be possible to circumvent the five or six object limit - possibly by rotating which buildings are tracked every second or so.

Usability Improvements

Purely digital interactives usually operate in the realm of two senses - sight and sound.  Sim Pueblo offers a third sense - feel - because the buildings are tactile.  However, much of the educational information about life in Kuaua pueblo is unavailable to people who are not sighted or who do not have high English literacy.  In fact, the users who are most likely to use Sim Pueblo at the site are small children who are often too young to read.  It would be good to add a narration component that can be toggled on and off.  This would not be difficult from a programming perspective, but it could be expensive and time consuming to hire and record a professional narrator.  This idea is currently in consideration by Coronado staff.

Another concern would be translating the interactive into Spanish and programming a language toggle mode.  New Mexico is approximately 47% Hispanic[12], and approximately 29% of Hispanics over the age of 5 speak Spanish at home[13].  This also would not be particularly difficult to program, but would require the Coronado Historic Site to hire a translator and possibly also have the tribal elders approve the translation.

As the user data sample size was rather small, it would be a good idea to add user activity logging functionality to Sim Pueblo.  It would not be able to differentiate between the sizes or ages of groups, but it would give a good idea of the length of game sessions.  In addition of being a measure of engagement, the length of each session could give an estimate of how much of the text is being read by the visitors.  It could also measure whether items were added to the mat, whether the instructions were accessed, and how many of the buildings were picked up by the building tracker.  This information would give more substantial measurement of how intuitive the UI is and how effective the object tracking method is.

Conclusion

The physical/digital fusion of SimPueblo presents visitors with a unique and compelling way to explore the lives of people who lived at Kuaua Pueblo hundreds of years ago.  It presented a number of difficult technical challenges because of the platform and user interface challenges from the fairly uncharted territory of using physical objects to interact with a computer.

I believed that when we purchased the HP Sprout, it would be a simple matter of training the Sprout to track the buildings and then programming a simple UI for it, but that was far from reality.  The HP Sprout solves many of the problems that were identified with Sim Pueblo 1.0, but has not solved all of them.  Even though the Sprout cannot track individual 3D objects, it can track 2D analogs of those 3D objects.  Unfortunately, while it can track 2D objects, it cannot seem to track all of them.  If we use the RealSense camera, we will be able to track objects in 3D, but we may not be able to do all nine buildings at one time.  The barrier between the physical world and the digital one is coming down, but slowly.  Right now there are no perfect solutions for a project like Sim Pueblo, but I believe there soon will be.  

Even with occasionally faulty object tracking, Sim Pueblo is an effective teaching tool that engages and delights visitors.  The primary goals of Sim Pueblo 2.0 were met: object tracking was achieved regardless of user contact and with only unobtrusive images on the buildings, and it can serve as a model for similar future projects for other New Mexico Historic Sites and beyond because we can track 3D objects using off-the-shelf hardware.  Most importantly for a museum interactive, the education content was brought to the literal front and center, and visitors gain a small glimpse into the everyday life of people who lived in New Mexico hundreds of years in the past.


Acknowledgements

I would first like to thank and acknowledge Craig Cassidy for dreaming up the idea for Sim Pueblo in the first place.  My first experience with the project was over the summer of 2015 where I worked with Craig on a presentation and UI storyboard for the Sim Pueblo 2.0 project.  We did research and came up with the idea to use the HP Sprout to accomplish the main goals of Sim Pueblo.  After that, Craig did the graphic design and did most of the illustration for the mat characters.  I would not have been involved with this project if it were not for Craig.

The staff at the Coronado Historic Site has been wonderful and pleasant to work with, and have given me excellent feedback on how SimPueblo could be improved.  Ethan Ortega’s enthusiasm and goodwill were instrumental in arranging the funding to get the project off the ground.

The 3D buildings were printed by Allie Burnquist who patiently wrangled the 3D printers to give us multiple copies of each building.  Bresdin O’Malley helped create the first round of tags, produce the shards, and give me much needed support during the spring semester.

I would also like to thank my advisor, Jonathan Lee, for all of his technical advice.  He also was the primary author for the paper for Museums and the Web 2016.

I received a lot of helpful feedback on the UI from Professor Miriam Langer, a coauthor on the Museums and the Web paper. She was instrumental in helping me promote the project.

I would like to thank all of my committee members: Jonathan Lee, Stan Cohen, and Doug Patinka for their help and advice during the course of the project.

Most importantly, I wish to thank my husband Chris for being so supportive throughout the process.  I couldn’t have done it without him.


Appendix I: User Testing Data

Below is a summary of the user testing data that I collected over three trips to the Coronado Historic Site.

Age range/role of participants

Did the visitor appear comfortable approaching Sim Pueblo?

1: Seemed unsure about approaching. 5: Enthusiastically approached

Did the visitor take time to read instructions before beginning interaction?

1: Did not read instructions, 5: Took time to read instructions thoroughly

Describe the visitor's initial interaction with Sim Pueblo (Did she/he pick up pieces right away, discuss with another visitor...)

Responses:

Did the visitor appear to understand the technical aspects of Sim Pueblo? (touch mat/touch screen/3D print placement & response)

1: No, they seemed confused, 5: Yes, they seemed to understand how to use all aspects

Did the user read any of the text?

1: Didn't read any, 5: Read all of the content

Did the visitor appear to understand the goals of the exhibit? (explaining aspects of ancient pueblo daily life)

1: No, appeared to not understand the point of the exhibit, 5: Yes, appeared to understand the point of the exhibit

How much time did the user spend on SimPueblo?

The possible options were: Less than one minute, 1 - 3 minutes, 3 - 5 minutes or 5 or more minutes

Appendix II: Software Class Diagram


Appendix III: Tag Images Used

All images in the table below are shown at actual scale.

Tag 1

Tag 2

Tag 3

Tag 4

Tag 5

Tag 6

Tag 7

Tag 8

Tag 9


[1] Lee, Jonathan, Simone Seagle and Miriam Langer. "A physical/digital playful learning environment for a New Mexico Historic Site." MW2016: Museums and the Web 2016. Published February 13, 2016. Consulted May 31, 2016.

http://mw2016.museumsandtheweb.com/paper/a-physicaldigital-playful-learning-environment-for-an-nm-historic-site/

[2] "Sprout JavaScript API Reference." HP Sprout JavaScript Framework Reference Guide. Hewlett Packard, n.d. Web. 31 July 2016. <https://sprout-developers.rssx.hp.com/documents/JS/SproutJavaScriptAPIDocs/>

[3] "Sprout SDK API Version 101." Sprout SDK API Reference for the WPF Extension for the C# Binding. Hewlett Packard, n.d. Web. 31 July 2016. <https://sprout-developers.rssx.hp.com/documents/CLR/>.

[4]For more information about the RealSense SDK, visit:

(Intel), Priyadarshini D. "Develop Immersive Experiences." RealSense - Developer Guide. Intel, n.d. Web. 31 July 2016. <https://software.intel.com/en-us/realsense/home>.

[5] This problem is noted on the Sprout Developers’ forum with no updates as of November 2015: https://sprout-developers.rssx.hp.com/forum/#/discussion/comment/109, however, a new version of the SDK released in Spring 2016 allowed access to the RealSense camera for developers.  Unfortunately, that was released only after I had gotten the buildings to be tracked successfully.  There is a longer discussion of this method in the Future Considerations section of this paper.

[6] Documentation and AForge.net download:

Kirillov, Andrew. "AForge.NET :: Framework." AForge.NET :: Framework. N.p., n.d. Web. 31 July 2016. <http://www.aforgenet.com/framework/>.

[7] For a primer on Turning Functions, visit: "Shape Comparison Using Turning Functions." Shape Comparison Using Turning Functions. N.p., n.d. Web. 31 July 2016. <https://sites.google.com/site/turningfunctions/>.

[8] Here is the link to the YouTube video that inspired the Tag Method: Edddisontoolkit. "Edddison on Sprout." YouTube. YouTube, 23 Nov. 2015. Web. 31 July 2016. <https://www.youtube.com/watch?v=f1Vx9t1zai4>.

[9] The Developer guide can be downloaded from at: https://sprout-developers.rssx.hp.com/documents/Sprout_Developer_Guide.pdf.  The relevant information can be found on page 158 of version 1.0.4 which I downloaded on 6/3/16.

[10] An explanation of object tracking and the location of the sample code can be found on the Sprout Developer website. "Sprout Developer Center." HP Sprout Developer Portal Object Tracking Comments. Hewlett-Packard, n.d. Web. 31 July 2016. <https://sprout-developers.rssx.hp.com/develop/tutorials/cpp-csharp-qml/object-tracking/>.

[11] Sprout Developer Guide, page 66.  See footnote 9 for download link.

[12] "State and County Databases." Pew Research Centers Hispanic Trends Project RSS. Pew Research Centers, 26 July 2011. Web. 31 July 2016. <http://www.pewhispanic.org/states/state/nm/>.

[13] Baker, Deborah. "Spanish Not 'enshrined' as Official N.M. Language." Albuquerque Journal. Albuquerque Journal, 3 June 2013. Web. 31 July 2016. <http://www.abqjournal.com/208492/spanish-not-enshrined-as-official-nm-language.html>.