Data Art
Data and Publics
Conceptual Project
Swapna Joshi
My personal experience with trees and seasons
In NYC
I am fascinated by the different seasons and the gorgeous colors vibrantly encoring the end of summer as the trees put themselves to bed for the long sleep of winter.
The lifecycle that a tree goes through when it goes through the fall , spring cycles is beautiful.
The science that goes behind the changing of leaves is also very fascinating.
Eg: Chlorophyll is not the only player in the fall leaf color game. Present in other leaves and trees are the compounds known as Carotenoids and Anthocyanins. As the Fall days begin to get shorter and shorter, the production of Chlorophyll slows to a halt, eventually giving way to the ‘true’ color of the leaf.
I have shared a few pictures that I have taken during the past year in different seasons on the next slide
Current Tools
I tried to look at the current tools that help travelers better time their trips to have the best opportunity of catching peak color for trees in a particular area
Current Tools
Thoughts on creating an Augmented Reality Experience
Get a reference of the real world
The renderings seen would be real like
Avoid looking at the map but the actual location
The power of being in a physical space yet experience the wonders of another augmented world which adds to your experience
Away from the static representation of maps
Experience the visual element of trees in seasons in realtime
Possible Renders of Trees in AR - Project of an unknown artist
AR Project by Jim Cantore
Jim Cantore loves trees so much he grew one in the AMHQ studio! Watch as he explains how leaves change color in the Fall in 3D Augmented Reality.
Project concept
This experience will encourage people to go out into the nature and know more about the species of trees and different seasons. This will bridge the gap between different species of trees and people’s knowledge on trees.
I want to create an experience of viewing the annual progressive changing of the leaves in different seasons in realtime.
My personal experience of collecting photographs and viewing them repeatedly could be replaced with this experience.
You can relive the memory of experiencing a season in the past or future.
Inspirational Imagery
Built a Quick Prototype using AR
AR Marker: Plant Image
Using Vuforia in Unity to Augment the Plant
The app recognizes the marker and augments it with coloured leaves
User Scenario 1 :�WSQ park - I want to know about how the tree looks like in different seasons � I want to see when the trees will change their color - point my phone at a tree - see the entire lifecycle - by interacting with the spinner or slider - option of getting extra information. By tapping on different options which appear if the user wants to spend more time in knowing about the seasons��The user can point at a whole row of trees or a single tree - whatever fits in the camera’s range��User Scenario 2 - A person is at home wants to know about which places to visit in the smoky mountains - � What all can the user select - a specific tree ,�choose a place - �1. map? Might not be the best option as you only see the rough scenario of the colors could be an option -�2. Choose a park, a specific area inside the park - areas and then apply see the actual photographic images and play around with the color slider to check the results� ����
Medium: Photoshop Mockups
A simple slider to experience another season after pointing the camera at a tree or a row of trees.
Click on experience a season
to change the current season
Slider appears - to change the season
Fall
Winter
Close the slider - go back to camera
Summer
Simulating the simple Interaction using Photoshop
GIF - (has some error while playing in the presentation please download from the link) https://drive.google.com/file/d/1jsE7OORMTEZdJsM7b63WRXkGiKNuvVnC/view?usp=sharing
Implementation -
of the concept
Figuring out the shape of the tree?
Can AR kit detect objects such as tree?
Core ML?
Getting the data set as API’s into Unity?
Will recognizing the tree species help?
I started looking at various data sets and talked to a lot of ITP classmates -Shivanku, Roy, Anastasis and professors - Rui to figure out the workflow to implement this concept.
Data source Google Maps
(figured that this data source might not work)
Mapbox.com (figured that this data source might not work)
Final Workflow after Research
Overlay/replace leaf renders using unity
Data sets to train the model
Integrate trained machine learning models into the app and create a core ML model.
ML workflow
+
Recognize leaves of the tree
Neural network 2
Integrate trained machine learning models into the app and create a core ML model.
Data sets to train the model
Data sets to train the model
Recognize the tree species
Neural network 1
'Moonrise' Full Moon Japanese Maple, 'Ruby Falls' Redbud, 'Evening Light' Japanese Snowbell, Smoke Tree, Acer palmatum 'Ukigumo’, Sourwood (Oxydendrum arboreum), Sargent's Cherry Tree (Prunus sargentii), River birch (Betula nigra), Flowering Dogwood (Cornus florida), 'Prairifire' Crabapple (Malus 'Prairifire’), Sweetgum Tree (Liquidambar styraciflua), Star Magnolia 'Royal Star' (Magnolia stellata), Eastern Hornbeam (Ostrya virginiana)
I tried researching on Tree species that go through the significant change in colours during the different seasons of the year. Here are the results
Birch , Elm , Maple, Oak
Some of the components of the NYC Opendata can be used
Recognize the species of the tree
Different parts of the tree data - recognize the leaves from the tree
One of the Data sets that can be used
Another Data Set that can be used to recognise the plant species using imagery