1 of 39

Philly Eats Analysis

November 2019

Lead with Experience.

2 of 39

Research

Overview of Experiment

Performance & Retention

Onboarding

Location Analysis

Browsing Behavior

Notifications

Restaurant Visits

Findings

AGENDA

3 of 39

Considerations

OVERVIEW OF EXPERIMENT

This app is different than a news app. Usage for this app is dependent upon wanting to dine out, and not about the daily news agenda.

This app is an experiment. Many elements of this app are custom and ad-hoc, but it also primarily leverages existing Inquirer content.

This app puts the location of a story closer to the center of the experience. How location factors into app habits is key to understanding usage and engagement.

Benchmarks are difficult to set for new experiences, but later on we reference general app usage and retention benchmarks based on Localytics and App Annie data.

4 of 39

The audience is representative

OVERVIEW OF EXPERIMENT

We have a representative audience size or *659 users.

The number of active users allows us to be 95% confident that the broader audience, Inquirer.com Food readers (review-only readers aren’t trackable), would use the app in a similar manner within a 5% margin of error.

Example: 41% of app users used a filter, so we can be 95% sure that 36-46% of all app users would also use the filters.

https://www.surveysystem.com/sscalc.htm

*This excludes: Lab staff, Inquirer employees (who submitted an Inquirer email address) and people who didn’t complete onboarding.

Avg. Monthly Inquirer.com Food Readers	Representative App Users Needed	Confidence Level	Confidence Interval (Margin of Error)
306,191	384	95%	5%

5 of 39

INDUSTRY RESEARCH

The Idea:

Like architecture content, restaurant reviews and recommendations are place-based and present a good opportunity to continue exploring the relationship between a story’s location and its audience—and how we can use those insights to build new and better products for local news.

6 of 39

Three Part Research & Analysis Plan

RESEARCH

Competitive Analysis of “Food finding” apps currently in market
In-person surveys about food finding habits, needs and wants
Google Analytics analysis on Inquirer.com food section content

Link to full analysis

7 of 39

Competitive Analysis

RESEARCH

Methodology: Review positive and negative reviews of existing food finding apps, as well as features and functionality to determine differentiating value propositions for a new app.

Findings:

Keep: Location based functionality is important and used by ⅚ apps reviewed.
Keep: Provide hyperlocal reviews relevant to where a user is while using the app.
Consider: Focusing less on making reservations since many apps specialize in this.
Enhance: Prioritize trustworthy professional reviews from Inquirer food critics over user generated reviews.
Enhance: Provide the option to read things later, or save restaurants to a list, and be alerted the next time they are near a place they’ve saved to facilitate easier exploration.

8 of 39

In-Person Surveys

IRESEARCH

Methodology: Surveyed 30 people to learn more about food finding methods, opinions on existing apps, and opinions on possible features and functionality.

Insights:

People said they go out to eat/drink 2-3 times a week for lunch or dinner based on recommendations from friends and family.
People tend to go to restaurants in their area (home or work), but will travel to locations such as Center City for more dining options.

App Feature Findings:

Only half were completely satisfied with their current food finding apps.
People expect filtering, reviews, near me suggestions, and reservation functionality for food finding apps.
Opinions about nearby notifications were mixed:

Those who would not want notifications were concerned about them being annoying and/or unsolicited so not relevant at the moment they receive it.
Those who thought notifications would be helpful cited possible time savings and added convenience, as long as the frequency was smart.

Respondents aged 18-65 with the highest number of respondents (8( falling into the 25-34 age bucket.

9 of 39

Inquirer.com Web Behavior Analysis

RESEARCH

Methodology: Use Google Analytics data to analyze current popularity and engagement of Inquirer.com professional food reviews.

Findings:

Ages 25-34 is the largest demographic reading food content, split evenly between males and females.
Food content does not necessarily have to be new to be read with about ⅓ of food pageviews coming from articles/reviews published over a month before analysis was conducted.
Food articles have high online engagement with an avg. 10% higher completion rate than the overall site average and a high volume of comments and shares.

10 of 39

OVERVIEW OF EXPERIMENT

The Result:

Philly Eats: Designed to provide an easy and more organized way for people who read and rely on professional reviews to decide where to eat and drink

11 of 39

Strategic Goals

OVERVIEW OF EXPERIMENT

Provide an easy and more organized way for people who read and rely on professional reviews to decide where to eat and drink to make decisions.

Observe usage and engagement trends to identify new ways to drive retention and satisfaction of food review readers

Begin to scope the larger opportunity associated with features and designs that respond to someone’s location/location needs.�

12 of 39

DATA & INSIGHTS

Performance

& Retention

13 of 39

App installs had a strong start with promotions on Twitter and in the Billy Penn newsletter, then leveled out to ~15 new users per week. New downloads and engagement from current users have a huge spike with the announcement of 2019 Dining Guides and app print promo.

DATA & INSIGHTS

Social and Print promotions are driving app downloads and engagement from returning users.

Source: Google Analytics

659

Active App Users

1,441

Total App Opens

2.2

Avg. Opens per User

Twitter promotions by Inquirer accounts/reporters, Colin Weir, Adam Erace, & Craig LaBan

Twitter promotions by David Arken & Lexi Belcufine

Billy Penn Newsletter

Twitter & Print promotions for 2019 Dining Guides

14 of 39

55% of users abandon the app after using it once; twice as high as the typical 25% app abandonment rate*. This may be because people familiar with the Inquirer download out of curiosity but didn’t intend to use it.

Those who use the app more than once average approx. 1.2 app opens per month.

DATA & INSIGHTS

There are 2 main cohorts of users: 1 time users and more frequent users.

Source: Google Analytics

Excludes users who do not complete onboarding sequence (7 total)

*2019 App Retention Benchmarks

15 of 39

30% of users are still active on the app a month after their initial open; only slightly lower than the typical rate of 43% app retention after 30 days in 2018, and slightly lower than that for non-game apps, which is in line with Philly Eats.

DATA & INSIGHTS

Thanks to the more frequent users, app retention is in line with benchmarks*.

Source: Google Analytics

Excludes users who do not complete onboarding sequence (7 total)

*2019 App Retention Benchmarks

16 of 39

DATA & INSIGHTS

Onboarding

17 of 39

The onboarding process is designed to capture permissions for:

1) Notifications

2) Location tracking

3) Email capture

These permissions allow the user to receive notifications for nearby restaurants, and allows additional tracking capabilities for the lab (restaurant visited, location of user, etc.)

98% of users have completed onboarding

DATA & INSIGHTS

3 Step Onboarding / “Welcome”

Source: Google Analytics

18 of 39

Just under half of users submitted their email during onboarding (about 300 users).

Collecting emails allows for additional communication with users, such as sending surveys, for additional learnings.

DATA & INSIGHTS

50% Email capture

34% Location capture

50% Notification capture

What permissions have users given?

Source: Google Analytics

Email Submission

19 of 39

DATA & INSIGHTS

Location Analysis

20 of 39

DATA & INSIGHTS

App opens are concentrated in Center City, where there are more restaurants; but a lot of activity comes from the suburbs as well.

Source: Google Analytics

App Opens (8/06 - present)

21 of 39

The majority of restaurants viewed are in a different neighborhood than the user’s current location.

DATA & INSIGHTS

While there is a cohort of users looking at restaurants in their current neighborhood, 86% of users who are interacting with restaurant detail pages are doing so in a neighborhood other than where they currently are.

Source: Google Analytics

Based on 170 unique users who are interacting with restaurant detail pages (view, save, call, reservations, website, full review)

% of Users Engaging with Restaurant Detail Pages

Location of Restaurant

22 of 39

FINDINGS

Notifications

23 of 39

Restaurant opening notifications have an average open rate of 9.5%

DATA & INSIGHTS

11 opens (7.5%)

18 opens (12.3%)

13 opens (8.9%)

25 opens (11.2%)

24 of 39

Nearby notifications have an avg open rate of 3.4%

DATA & INSIGHTS

This is the result of 2 opens from 58 nearby notifications sent to date.

Spice C: 2 opens

Chinatown

12 notifications sent

Rittenhouse

8 notifications sent

Old City

9 notifications sent

25 of 39

FINDINGS

Restaurant Visits

26 of 39

A third of users who save a restaurant visit it

DATA & INSIGHTS

5% of users (17) have been located at a restaurant however this number may be underreporting the amount of users actually visiting restaurants after using the app due to disabled background location tracking (which is necessary to be always on to track this data point).

Rittenhouse is the neighborhood with the most visits

9.1% of users who view a restaurant visit it

27 of 39

FINDINGS

What did we learn?

28 of 39

What did we learn?

OVERVIEW OF EXPERIMENT

Provide an easy and more organized way for people who read and rely on professional reviews to decide where to eat and drink to make decisions.

Goal 1:

The app is successful at driving users to restaurant reviews.

Over 60% of app users view at least one restaurant with the average at 3.3 reviews opened per user.

Guides have grown in popularity after the release of new content.

40% of users are viewing guides, with the most popular being “Best of” type content (Top 25, Four Bells, etc.) vs. cuisine based guides.

Filters are useful to users, specifically for neighborhood searches.

30% of users are utilizing filters at least once, with the neighborhood filter being included in 64% of searches.

Local browsing is the most important feature to users.

While filters and guides are important, the most popular way users are browsing content is by swiping through the carousel of restaurants in their immediate area.

29 of 39

What did we learn?

OVERVIEW OF EXPERIMENT

Observe usage and engagement trends to identify new ways to drive retention and satisfaction of food review readers

Goal 2:

Philly Eats has a higher than average amount of one time users.

This is possibly due to interest in trying the product due to their familiarity with the Inquirer with no real intention to be a regular user.

Other than one time users, Philly Eats is an occasional tool for people.

Based on research conducted, most people get restaurant recommendations from friends/family making Philly eats more of an occasional tool, as seen in the trend of users opening the app approx. 1.2 times per month to date.

Announcement notifications are well received.

50% of users authorized Philly Eats to send notifications to their phones, of which we are seeing a 9.5% open rate. These types of notifications should continue to be used selectively as to not over burden a user.

30 of 39

What did we learn?

OVERVIEW OF EXPERIMENT

Begin to scope the larger opportunity associated with features and designs that respond to someone’s location/location needs.�

Goal 3:

Nearby notifications have not had the same success as announcements with a open rate under 4%.

To date, only ⅓ of users authorized their location to be tracked at all times (required to send nearby notifications) and only 2 out of 58 notifications were opened. These metrics could possibly be improved by putting more emphasis on the nearby notifications functionality during onboarding, but are not the main value proposition of the app.

Default map view of 5 closest restaurant is successful in showing users what they are interested in.

Only 10% of users are manually re-adjusting the map from its original position vs. the 70% that are swiping through the carousel of restaurants in view.

31 of 39

FINDINGS

Other Considerations

32 of 39

Notes

Onboarding results are based initial set up settings, a user can change settings in app or on their phone at any time.

Tracking update is in progress to capture settings at every app open to track changes here.

iOS 13 launched a new feature that requires a user (who enabled location to be always tracking) to reconfirm that setting, likely causing a drop in users with the “always allow” functionality enabled.

Having this setting as “always allow” is necessary to track location visits and send nearby notifications, so this metric is likely under reporting conversion success of the app in terms of driving users to restaurants

DATA & INSIGHTS

Source: Google Analytics

33 of 39

Lead with Experience.

34 of 39

Significance Significance vs. Representative Audience

OVERVIEW OF EXPERIMENT

Statistical Significance	*Representative Audience
Measures if the result is “real” and not due to chance and is specific to an experiment where there is a control group and a test group where all conditions are equal except for one variable (also known as A/B testing) Should be used for A/B testing when thinking about changing a product and should measure significance before rolling out to the full population	Used to determine how many people you need to observe in order to get results that reflect the target population as precisely as needed Confidence Level: tells you how sure you can be, expressed as a percentage. Most researchers use 95% confidence as a best practice Confidence Interval: plus-or-minus figure to apply to the metric which would encompass the entire population

*Relevant metric for Philly Eats Experiment

35 of 39

“

”

Philly Eats

OVERVIEW OF EXPERIMENT

We all believed that local news could compete with other profitable and popular digital apps and services in the restaurant space, but on a local level. We believed that the quality, breadth and depth of The Inquirer’s coverage offered a competitive advantage over products offering a slew of user-generated reviews.

Sarah Schmalbach

36 of 39

Philly Eats is showing 34% of users selecting “Always Authorize Location” to date. During the first wave of downloads, this was around 19%, then grew to 53% after the 2019 Dining Guide launch and print promotion, but still lower than HERE at 62%. In addition, 50% authorized notifications (vs HERE at 75%).

DATA & INSIGHTS

Location and notification permission rates are lower than HERE, but they aren’t primary to the experience.

Source: Google Analytics

37 of 39

Our hypothesis for this behavior is related to the messaging of the purpose of the app. With the HERE app, the main value prop to downloading is that you will be notified when passing noteworthy architecture, so location tracking and notifications are inherently required.

For Philly Eats, the app is positioned as more similar to Yelp, an exploratory way to find new restaurants with trustworthy professional reviews. Being notified that you are near a restaurant is a secondary benefit to having the app, prompting the lower authorization rates.

With the launch of the 2019 Dining Guides the nearby notification feature became more valuable (get notified while walking past a restaurant on the guide), explaining the boost in authorization rate.

DATA & INSIGHTS

Source: Google Analytics

Philly Eats Description

HERE Description

38 of 39

FINDINGS

Browsing Behavior

Highlights

39 of 39

Swiping: Swiping through the carousel is by far the most popular way to browse content with 70% of users utilizing at an average of 10 swipes per session.

Map View: Only 10% of users adjust the map size/location. The low usage of map adjustments indicates an effective default map setting as showing 5 location pins.

Filters: Used by 41% of users, neighborhood is the most popular (supports the main value of the app - explore options based on location)

Guides: Guides were being used by 30% of users before the launch of the 2019 Dining Guides, then jumped up to 55% since launch. In addition with the launch of these guides, filter usage dropped to 30%, indicating two distinct ways the app is being used.

Cuisine: Italian is the most popular searched-for cuisine (consistent with Inquirer.com where "Italian" is one of the top food or drink mentions in headlines.

Bells & Price: There’s a lot of interest in low price, high quality restaurants.

Search: About a third of users are utilizing the restaurant name search, however almost 60% return no results (79% of searches are restaurants not in the app yet)

DATA & INSIGHTS

Source: Google Analytics