Published using Google Docs
BobChat Report (Public)
Updated automatically every 5 minutes

BOBCHAT REPORT

INTRODUCTION        2

PREMISE & GOALS        2

TIMELINE        3

GUIDES & POLICIES        6

ULTIMATE INTERN GUIDE        6

OTHER POLICIES        6

CONVERSATIONS REPORT        8

DATA COLLECTION        8

ANALYSIS        10

POTENTIAL        16

WHY CHATBOTS        16

BOBCHAT'S FUTURE        20

SCOPE        20

PLATFORM        21

BRANDING        22

STAFFING        23

FINAL CONSIDERATIONS        23

APPENDIX        25

DETAILED TIMELINE        25

TRANSCRIPT EXAMPLE        28

INTERNSHIP DESCRIPTION        33

PLATFORMS GUIDE        35

ENGAGEMENT GUIDE        42

STYLE GUIDE        47

INTRODUCTION

This report, compiled in Summer 2020, details the history, development, and usage of BobChat — a Facebook Messenger chatbot created by the Office of Student UX, Engagement, and Technology as a digital concierge and resource hub for the NYU community — in the three years from its inception to its current stage. It also outlines the future potential of the chatbot in new and improved iterations, highlighting its necessity to the NYU community as an accessible, affordable, innovative way to improve their experience with the University.

We are proud that BobChat was built by a small team of student interns, making it one of NYU’s most cost-efficient, all-University digital initiatives to-date: costing less than $500 to build with monthly maintenance costs of under $100 (aside from student staffing expenses).

 

PREMISE & GOALS

BobChat originated from conversations with NYU Student Government members who approached NYU’s Office of Student UX (then called NYU’s Office of Interactive Media) to propose a new platform for students to provide feedback to the University.

At first, Student Government envisioned a forum where students could submit issues and upvote or downvote each other's submissions, bringing the largest or most visible issues to the top so that the University could respond to or fix them. Our office saw some setbacks in this idea — mainly the potential for quickly-solved but more mundane issues (like a broken printer) to be downvoted or ignored, while more complex and difficult to solve issues (like tuition affordability) might crowd the top of the platform.

In discussing the goals of this platform, we realized a chatbot could meet feedback needs, while offering a broader scope of features: a navigational menu, supplemented with artificial intelligence, allowing users (students or others) to explore the University at their own pace; a live chat function, connecting users to knowledgeable staff when the chatbot could not find the correct answer; and, a familiar interface that most students already used regularly, reducing friction to increase adoption.

As we began development, our vision was to make BobChat a convenient, engaging, all-inclusive point of contact for users — prospective, newly-admitted, or returning students, as well as their families — to navigate the University, discover resources, leave feedback, and have their questions answered 24/7, anywhere in the world.

TIMELINE

While a detailed timeline of meetings and attendance is available in the Appendix, the following is a general breakdown of BobChat's development:

DECEMBER 2016

Our office first met with NYU’s Student Government to discuss a feedback platform. Meetings continued over the course of the Spring semester.

SUMMER 2017

We hired our first chatbot development interns (a team of two).

FALL 2017

BobChat's first iteration went live on the NYU Student Government Facebook page on August 27, 2017. It was not broadly advertised, so that our team could keep refining and expanding the chatbot with our limited capacity. We had one intern (located in NYU Berlin) fully devoted to the bot, and another eight interns in New York assigned to a different project, who occasionally assisted in brainstorming and providing feedback.

SPRING & SUMMER 2018

In preparation for a wider launch, we began a months-long process of meeting with 25 partner offices to review the BobChat content related to their departments. We held a total of 27 meetings, mostly between March and August 2018, with a few stretching into April 2019, due to scheduling conflicts. During this time, we also moved BobChat into its own Facebook page.

FALL 2018

We expanded the BobChat team to eight interns, divided into a "technology" role, focused on refining the AI and adding content, and an "engagement" role, focused on brainstorming content expansion and directly interacting with users. We solidified our protocols — meeting with IT’s security team and general counsel, as well as wellness groups — and created a marketing strategy in preparation to launch during the late Fall 2018 or early Spring 2019 semester.

SPRING 2019

We officially announced BobChat to the NYU community, sending a direct email to an audience of about 40,000 students located in New York on February 12, 2019. The response was overwhelming, with nearly 1,000 users contacting the bot within the first 24 hours.

Some interesting stats from the first 24 hours of BobChat:


The rest of our advertising came from our partner offices, with whom we shared a media kit, on-campus digital signage, and a tabling event during Weekend on the Square. Having mostly combined the "technology" and "engagement" roles, our intern team took shifts to monitor live chats from 9AM to 9PM ET on weekdays (except Friday, when shifts ended at 5PM ET).

SUMMER 2019

With a smaller intern team of two interns, we continued refining the bot and planning more marketing materials for the Fall semester, while we awaited decisions on ongoing budget.

FALL 2019

Without a larger budget and capacity, we did not advertise the bot to new students. Our two interns this semester were located in Abu Dhabi and Shanghai, leaving one full-time staff member available for live chats during business hours (9AM to 5PM ET).

SPRING 2020

We put BobChat on hold on January 31, 2020. As of Spring 2021, all of its content is still available, but not kept up to date. We included a notice so any new user, or returning user accessing BobChat's main menu, is aware of this.

GUIDES & POLICIES

As BobChat's content and features grew in scope and complexity, we established a number of protocols to ensure consistency for the sake of our team, audience, and product. Some guidelines helped us to protect our users' safety and privacy; some ensured the bot's content was up-to-date, engaging, and accessible; and others helped us manage our team and keep everyone on the same page.

ULTIMATE INTERN GUIDE

Below are summaries of the different sections in our Ultimate Intern Guide, available in full in the Appendix.

PLATFORMS GUIDE

Details internship communication platforms and practices, task workflows to create and edit content, artificial intelligence, and automation features for BobChat.

ENGAGEMENT GUIDE

Defines our customer service best practices, explains procedures and workflows, and offers troubleshooting for common issues.

STYLE GUIDE

Outlines BobChat's voice and tone, as well as grammar, spelling, and formatting conventions that we expect interns to adhere to when writing content.

OTHER POLICIES

OFFICE MEETINGS & TRANSCRIPTS

In preparation for our launch, we met with offices across Student Affairs and other student-related divisions. In these meetings, we explained and demonstrated the bot, provided offices with transcripts, and requested their partnership in creating a bank of keywords and frequently asked questions to strengthen BobChat's AI. Transcripts listed all the content in a particular section of BobChat — text, GIFs, buttons, and so on — which we compiled into documents for partner offices to revise and approve. Later on, we moved these transcripts to the BobChat Knowledge Base. An example of our initial transcripts is available in the Appendix.

TEAM

As mentioned in our timeline, our team went through several changes over the course of BobChat's development. We had between one and eight interns fully devoted to the bot at a time, and their roles changed across semesters. Some semesters, we split the team roles between technology and engagement, but later found this division unnecessary — our latest internship description is available in the Appendix. A larger team was imperative for our live chat coverage since our students could not work long shifts, but this also came with challenges: it was harder to keep track of content changes, and ensure their consistency, when eight people had access to the bot. Later in this report, we propose our ideal team makeup based on the lessons we learned.

LIVE CHAT COVERAGE

It was core to our mission that users would be able to talk to a human outside of NYU offices' business hours — and ideally, 24/7. This flexibility frees users to find information at their own pace and on the go, taking full advantage of chatbot technology and lessening the workload of NYU offices during their business hours. Due to our budget capacity, we decided to monitor live chats between 9AM and 9PM ET between Mondays and Thursdays, and between 9AM and 5PM ET on Fridays.

CONVERSATIONS REPORT

DATA COLLECTION

Compiling conversation analytics was one of our team's biggest challenges. Neither Chatfuel nor Facebook allowed us to fully export the conversation data we were looking for, like a transcript of messages sent and their timestamps. Because of the complexity and size of BobChat, it wasn't enough to simply see the top five messages sent to users, or the number of new and returning users. BobChat does not have a simple sales funnel (or any, for that matter) — it has 1500+ units of content, divided into 44 groups. We needed to see all messages sent to, and by, users so that we could sort them into subject areas, levels of accuracy, reasons for error, and so on.

Over Spring 2020, as we paused BobChat, we took the opportunity to manually create the analytics report we wanted. Unable to quickly export conversations, we read each message received and sent between February 12, 2020 (our official launch date) and December 31, 2020. To clean up the data, we decided to only capture serious, NYU-related questions — filtering obvious trolling, joking, or inappropriate messages, as well as messages that were not in any way relevant to the NYU experience. This amounted to a total of 6,427 messages, which we then sorted into the following categories:

ANALYSIS

Below are the key takeaways from our data collection and analysis. All data and graphs can be examined in detail in the BobChat Conversations sheet.

ACCURACY

These numbers show us that BobChat has a long way to go in terms of AI and content. It should be noted, though, that adding more content would make the AI even more of a challenge. It might be feasible, and even preferable, to limit BobChat's scope in order to increase its accuracy and efficiency — but in that case, users need to be properly educated on the bot's limitations to avoid becoming frustrated.

GROUPS & RESPONSES

These volumes show us that most users were interested in quick and convenient access to campus-focused information — how to get there, where to eat, and so on.

Given that Entrepreneurship and Shanghai had a low volume of messages, their accuracy is less impressive than the other two. Majors had a medium volume, but its accuracy fell down to 80% when filtering for freeform messages only. Since Public Safety was the most popular group by far, its 96.97% overall accuracy and 93.98% freeform accuracy stand out as a success.

Public Safety's success is likely due to the fact that bus schedules are very simple to program. "Bus X," "route X," or "shuttle X" as keywords are both short (meaning they are easily identifiable to the AI) and commonly used (meaning a user is more likely to type "Bus A" than "Bus to Tandon," which would be more difficult to program).

This does not mean a new iteration of the bot should necessarily get rid of this group — users regularly engaged with and were interested in small talk. As part of a quality conversational chatbot experience, small talk needs robust programming that our team did not have the expertise or tools to build. If the conversational aspect is not a future priority, small talk should still exist in more limited amounts.

USAGE OVER TIME

USER BEHAVIOR

POTENTIAL

In this section, we will discuss the potential of chatbot technology to improve the NYU student experience. We will end the report with our recommendations for the future of BobChat — including expected challenges and setbacks, lessons learned, and opportunities for growth.

WHY CHATBOTS

As BobChat slowed down and we began to think about its future, the book The Age of Intent by P.V. Kannan provided a renewed articulation on the importance and potential of chatbot technology in creating better customer experiences. In this section, we will discuss the main points of the book and how they relate to the NYU student experience.

In the book, Kannan argues that virtual agents (chatbots) with a conversation-driven approach will rise in prominence in the field of customer experience, by virtue of solving or alleviating a variety of user concerns:

"Virtual agents improve:
1. Consistency—give the same right answer every time.
2. Uptime—make service available 24/7.
3. Capacity—scale up to serve customers quickly, even during peak service periods.
4. Speed—reduce time spent waiting for human agents.
5. Productivity—help human agents deliver smarter and better service.
6. Intelligence—generate new insights by analyzing aggregated service interactions.
7. Channel independence—consumers can use voice or text chat and get the same answers from the same bot." (53-4)

All of these factors are vital in NYU's mission to better serve students: in a decentralized and ever-growing institution, consistency, capacity, speed, and productivity are increasingly challenging to improve. A larger student body (and larger pool of prospective students, as NYU's prestige increases) will only put more strain on University systems. In a cutting-edge university, both named after "the city that never sleeps" and comprised of 14 global locations spanning 9 different time zones, 24/7 service is not only logical: it should be intrinsic to our global mission. As Kannan explains, a chatbot can solve these problems for NYU while improving the productivity of its employees and the intelligence of its strategies.

Arguing that the shift to conversation strategies is inevitable, Kannan compares the rise of chatbots to that of websites and mobile applications in previous decades:

"The main cultural hurdle is prejudice: the belief that human agents can always deliver a better customer experience than virtual ones. [...]
The reality is, of course, that consumers would often prefer to interact with a machine rather than a human—if that machine can get to the answer quickly. This is why the web took off in the '90s and apps became successful in the past decade. Consumers preferred clicking on their computers or tapping on their phones to see when a package would arrive or what their bank balance was—the result was quicker and more dependable." (140)

NYU is not exempt from this — and, in fact, since our customers (students) lean much younger than the general public, they will seek and demand this kind of experience earlier than other groups might. While parents and older alumni might be happy to continue calling offices, current and future students are less and less likely to feel the same way.

As Kannan mentions, one of the concerns regarding chatbots is the prejudice or fear that they will replace human labor, and do a poor job at it. To address this, Kannan states that "[t]here is no replacement for interactions with an agent. The level of empathy is not there. But what we can do is eliminate the simple pain points." (48) By eliminating or at least reducing the bulk of calls regarding simple, frequently asked questions, a chatbot would free NYU administrators to help with more complex issues that do require human empathy and nuanced knowledge. According to Kannan, in companies that implemented chatbots, contact center agents are "actually happier; they've been freed of the more roting customer requests." (51) Rather than having busy administrators, we should aim to have focused administrators, with enough energy to put their skills into helping students when technology cannot.

Furthermore, chatbots offer an entirely new set of tasks and opportunities for human jobs, in the form of supervising and refining the bot:

"A human agent might supervise eight or ten chatbot conversations, far more than a human agent could handle if they were doing the talking. When a virtual agent gets stuck, the more experienced human can step in and solve the problem.
But the human has one more job: to tag the customer's intent, which the virtual agent may have missed. This information gets fed back into the virtual agent system and makes it a little smarter, so next time it can handle a similar interaction on its own." (102)

… as well as collecting and analyzing data:

"Every conversation makes companies smarter about their customers. [...] Every conversation is a communication generating data that your competitors are missing out on. As a result, every conversation will improve the strength of your connection with these customers." (199)

By creating these new opportunities, NYU can retain its workforce, simply replacing the menial, repetitive tasks that take up so much of administrators' time with more meaningful and stimulating tasks that advance strategic goals. All of this can be done while serving a much higher volume of students, creating more efficient and sustainable operations all around.

Another common concern is the potential for glitches or errors. As Kannan puts it:

"All virtual agent systems have holes in them. Sometimes, they guess the wrong intent. Sometimes, they can't figure out the intent at all.
This is not a failure. Even if the virtual agent can't figure out the intent half the time, it's still going to make the whole system more efficient. Half the time, customers will quickly and consistently get what they're looking for. The other half, they'll quickly get handed over to a human agent who can solve the problem in the usual way.
But virtual agent systems aren't static. They can improve." (41)

Humans are not immune to error, either — and perhaps a chatbot's mistake is more easily tracked and fixed than a human's. Anecdotally, we found that users were forgiving of simple AI mistakes, frequently trying to phrase the prompt a different way in order to get the right answer. By virtue of the digital literacy inherent in a generation born into the Internet era, current and future students know to adapt their behavior to a specific digital medium — using shorthand syntax for Google searches, for example. As long as the technology keeps improving, and offers an avenue to find the correct answer (as BobChat did, by offering to connect the user to a human), errors should not be an argument against innovation.

As it stands, the student experience of calling or emailing offices, whether or not they provide the correct information right away, is also not without flaw. Students often have long wait times and are bounced around a variety of offices until they are connected to the right person. The only way to avoid this, from the student's perspective, is to have preemptive knowledge of NYU's complex organizational structure, knowing exactly who to reach ahead of time. This presents an added set of challenges for already marginalized student populations like first generation or international students. As an institution, some seem to believe that this is an unfortunate reality of any large university and that nothing can be done about it. But a centralized customer service system can take the burden off of students by figuring out who they should contact, and even offering to do so on their behalf. Kannan points out that "[c]ustomers don't know or care who in a company owns the individual experience of billing, onboarding, service calls, and so forth. From their perspective, these are all part of one and the same journey." (171) In the same vein, a student with a tuition-related question does not care about the difference between bursar and financial aid — they simply are worried and need help. By maintaining a decentralized customer service approach, NYU is in fact providing more and more opportunities for student dissatisfaction:

"Imagine, for example, a customer … who interacts with an app, an IVR system, a call center, and a website in the course of solving their problem. For the sake of argument, assume that each of these channels has a 90% chance of delivering a satisfactory experience. The chance of the whole chain of interactions delivering satisfaction is 90% x 90% x 90% x 90%, which is only 66%. So while the executives in charge of each touchpoint imagine themselves to be doing a bang-up job with 90% satisfaction, the customer is only satisfied two-thirds of the time. This is why a focus on individual touchpoints is inadequate to understand satisfaction in journeys that cross channels." (172)

Therefore, NYU's approach to serving students should simplify and remove steps from the student's perspective, offering quick answers to simple questions and fully guiding them through complex ones. A chatbot can achieve that by providing constant, convenient, consistent service — any time of the day, anywhere in the world, no matter how many times the question gets repeated.

Of course, simplifying these processes is an incredibly difficult task. As Kannan puts it, "[E]very company wants to give customers what they want with less friction. What gets in the way? Inflexible architecture and systems that don't connect" (155). To build the ideal seamless chatbot, NYU would need to integrate existing systems like Albert, NYUClasses, ServiceLink, Engage, and its website — to name a few. "Until the APIs on a company's key systems are in place and working dependably, there's no practical way to connect a virtual agent to those systems" (135-6). Despite the challenge of the task, Kannan makes it clear that it is a step institutions need to take in order to avoid technical debt — the "accumulation of problems due to failing to invest in keeping corporate systems up to date" (22).

"[I]n the end, companies that don't adopt flexible architecture and APIs into their systems will fall behind on all technology channels, not just virtual agents. The march of progress and rising consumer expectations will force companies to open up and modernize their systems so that virtual agents can get access. The alternative is an increasingly inflexible and disconnected company that can't move at the speed its customers demand." (188-9)

Students can see and feel the technological advances made outside of the world of higher education — and yet universities fail to catch up, all while continuing to increase tuition costs. It is only logical, then, that students will become increasingly frustrated with their experience, and begin to question its worth. This can easily be followed by a decline in school prestige, alumni pride, employee morale, and so on. In order to maintain its image as an innovative university, NYU must innovate.

The Age of Intent ends with a warning that we believe NYU, as well as all universities, must take to heart:

"We're at the very beginning of this profound transformation. It's hard to know just how fast it will get here, exactly what shape it will take, and how it will change the worlds of business and human interaction. But make no mistake: the age of intent is upon us. And unless you're preparing for it, you're going to be left behind." (208)

BOBCHAT'S FUTURE

There are several ways in which BobChat can exist to improve the NYU student experience. Taking into account the challenges our team faced, and the opportunities we see ahead, we have divided these options into four different categories: scope, platform, branding, and staffing.

SCOPE

We advertised BobChat as answering "all things NYU" — while the content of the bot itself was limited, we believed the phrase could point to the fact that our team would attempt to answer any and all NYU-related questions via live chat. Not properly defining the scope of content likely led to some confusion between our users, who might have expected the bot to know detailed information about academic departments, for example. Future iterations should take greater care in setting the scope as well as thoroughly educating users on the chatbot's limitations. Some options we see within the scope category are:

ALL THINGS NYU

From a user's perspective, the best chatbot is the one that can answer it all. It should answer questions across academics, student life, campus services, etc, on its own. It would be able to offer detailed, personalized information, down to the individual student's perspective: it could immediately answer questions like "What is my GPA?" or "How many credits do I have left in my Spanish Studies minor?"; or even perform tasks like "Change my preferred name to Jane" or "Update my phone number to (555) 123-5678."

In practice, however, this ideal becomes extremely challenging, if not impossible. It would require AI capable of understanding the complexity and nuances of the NYU experience — the difference between schools, programs, global sites, enrollment statuses, student visa types, and so on. IT would also require the technology to access and store our students’ personal information (N number, GPA, etc). The Age of Intent addresses this problem, warning that "if you wait for the 360-degree view of the customer to be complete—or try to design a chatbot that solves all problems—you'll never live to see any practical benefits." (159). Given the complexity of the task, without the guarantee that the AI would be able to handle these nuances, we would not suggest a universal chatbot solution.

BASIC INFORMATIONAL

This option would provide a wide scope with limited depth. A chatbot following this structure could clarify NYU's institutional complexity — explaining different offices and services, providing summaries, defining terms, and so on — without necessarily creating new content or performing any tasks. This bot could link to existing articles and pages on ServiceLink or the NYU website, simply helping connect users to the information they are seeking.

TASK-ORIENTED

This option would either perform basic-level tasks ("Change my preferred name to Jane"), or be able to instruct the user on the step-by-step of more complex ones ("Submit my $50 student reimbursement") — or a combination of both. Given the range of complexity, this chatbot may require a limited scope by subject or department, or limited navigation (button-only) so that the user immediately understands which tasks are available.

CONTACT AND FEEDBACK

This option would focus on delivering customer service and identifying pain points. It would collect feedback, help triage who the user needs to contact, and offer to do so on their behalf. This could also be the least robust AI option, offering to connect users directly to a live agent during available hours.

PLATFORM

Where a chatbot lives is just as important as what content it holds. The platform hosting BobChat can help inform which content is best suited for the bot — and it certainly affects how users engage with it. We see three potential platforms:

SOCIAL PLATFORMS

We decided to host BobChat on Facebook Messenger for a variety of reasons — we believed most students would already be familiar with the platform, and have it installed on their phones, reducing the friction of learning a new platform; we found it easier to build the chatbot using technologies supported by Facebook Messenger rather than creating our own; and, we did not have the access required to host it on existing NYU systems. Depending on the team behind the next iteration of BobChat, these factors may still be relevant. We have always been aware, however, that social platforms like Facebook and Whatsapp generate concerns over privacy that may grow stronger. Because of this, we would suggest avoiding these platforms, unless there is not enough institutional support or budget to build a chatbot elsewhere.

NYU WEBSITE

NYU's public website would be the most visible platform for a chatbot, available to the widest audience — including prospective students and parents who may not have access to other NYU systems, or who may not be familiar with the NYU social media landscape. Because of this wide audience without the authentication system of collecting a student's NetID, hosting a very nuanced chatbot might not be possible — which is not a problem per se, given the scope options detailed earlier. Other benefits of this platform include increased user adoption and legitimacy, since the website is already familiar to students, and more trustworthy than third-party social media platforms. There may be, however, challenges in the actual hosting, requiring a lengthy development period with a great deal of institutional support and collaboration.

ALBERT

Albert would provide similar benefits as the website, but with a more limited audience (only those with NetIDs). If the user data can be collected and leveraged, however, this layer of authentication can provide increased security and allow for a more sophisticated bot. Albert would be the ideal platform for a task-oriented bot, fully connected to the system and able to not just inform the user, but provide seamless solutions to simple problems. This option would likely require significant changes to Albert's system as a whole, as well as extensive institutional collaboration.

BRANDING

While we created a unique brand identity for BobChat — complete with a color palette, personified mascot, and recognizable name — we realize these elements are not a necessity for the next iteration of the chatbot. The identity provides a sense of novelty and generates engagement, but it also opens room for opinions about the brand to distract from the content itself. Perhaps the discussion between users about whether the mascot is cute or ugly is the wrong type of engagement, or perhaps any engagement is better than no engagement.

The alternative is for the chatbot to have no brand or personification — similarly to the NYU website, which has no official or special name. This would allow for the chatbot to blend in with the NYU brand, showing it as an intrinsic part of NYU rather than a separate or temporary entity. While this lack of personification may be less engaging to some users, it might also lend the bot the legitimacy and trust that come with the NYU brand.

Since both types of branding have their pros and cons, the decision should be made with consideration to scope and platform. A unique brand might make more sense on social platforms than it does on the NYU website or Albert; or it might be more appreciated for a contact and feedback bot, where empathy and friendliness are more important than the clear-cut instructions of a task-oriented bot.

STAFFING

At its largest, our team consisted of eight student interns working a combined 80 hours a week, plus three full-time administrators (only one of which worked on BobChat for a majority of their time). This is simply not enough to properly staff a chatbot to be used consistently by even a fraction of the near 60,000 students at NYU — let alone prospective students, parents, and so on. We believe this to be the case regardless of scope, platform, or branding.

Ideally, multiple people would monitor conversations at all times, making note of AI mistakes and collecting data when they are not required to assist users in live chats. A large intern team will make staffing more affordable, but it presents other challenges — interns have less knowledge of, and experience with, navigating NYU's administrative structure, and their schedules usually only allow for short shifts, which can create handover issues in conversations. We would suggest, then, that experienced administrators are best suited to actually handle live chats, while student interns should compile data, strategize content, and offer solutions — given they have the unique perspective of the target audience. The ideal team consists of both groups, so that their strengths can complement each other for a better product. In terms of content, while a large team can suggest and even draft the material, the final approval should be limited to an administrator solely devoted to the project in order to guarantee consistency.

FINAL CONSIDERATIONS

Regardless of where, when, or how an NYU chatbot exists, two things are clear: first, that it should and must exist; second, that it will require substantial effort, budget, and cross-University collaboration to be the best it can be. University leadership must see the benefit and necessity of this technology for the sake of innovation, affordability, and service improvement — and it must feel invested in the product, fully supporting and empowering the development team. We have to go all in. This project has immense potential to improve the NYU experience, and we owe it to our students — present and future — to get it right.

APPENDIX

DETAILED TIMELINE

KEY DATES

OFFICE MEETINGS

  1. March 12, 2018: Academic Resource Center
  2. March 19, 2018: LGBTQ+ Center
  3. March 26, 2018: Athletics
  4. March 26, 2018: Global Spiritual  Life
  5. March 30, 2018: Moses Center
  6. March 30, 2018: Special Sessions
  7. April 2, 2018: Dining
  8. April 2, 2018: Global Services
  9. April 5, 2018: Global Programs
  10. April 6, 2018: Residential Life
  11. April 9, 2018: Student Government
  12. April 13, 2018: Student Health Center
  13. April 23, 2018: Bronfman Center
  14. April 23, 2018: Center for Student Life
  15. April 23, 2018: Islamic Center
  16. April 24, 2018: StudentLink Center
  17. April 30, 2018: Leadership Initiative
  18. April 30, 2018: Wasserman
  19. May 29, 2018: StudentLink Center (2)
  20. June 7, 2018: Bursar
  21. August 6, 2018: Registrar
  22. October 2, 2018: Public Safety
  23. October 4, 2018: Partners Meeting
  24. October 11, 2018: Undergraduate Admissions
  25. November 28, 2018: Graduate Admissions
  26. March 14, 2019: Entrepreneurial Institute
  27. April 19, 2019: Libraries

OTHER MEETINGS

  1. December 16, 2016: SSC Meeting
  1. Nick Jensen, SSC
  1. May 3, 2017: Student Government Advocacy // Interactive Media
  1. Alex Bao, Max Pau, Nick Jensen, and Sarth Desai
  1. June 22, 2017: NYUHelpDesk // Student Government
  1. Alex Bao, Max Pau, and Nick Jensen
  1. July 20, 2017: Chatbot Lunch
  1. Alex Bao, Max Pau, and Nick Jensen
  1. October 4, 2017: BobChat Meeting
  1. Max Pau, Nick Jensen, and Tina Solo
  1. July 9, 2018: Security Check-In for Student Affairs Chatbot
  1. Erin Callihan, Nick Jensen, Angela Chen, John Ming Lee, Kerri J Tricarico, Lec D Maj, Meenakshi Baker, Sarth Desai, ac7659@nyu.edu
  1. September 21, 2018: BobChat Wellness Convo
  1. Zoe Ragouzeos, Erin Callihan, Nick Jensen, Tina Solo, Mahalia Thomas
  1. October 9, 2018: ServiceNow for the Office of Interactive Media, Student Affairs
  1. Erin Callihan, Nick Jensen, Katelynn Burns, Lizz Duke, Sandeep Gupta, bkc1@nyu.edu
  1. October 18, 2018: Test-a-Palooza
  2. October 30, 2018: NYU Connect and BobChat
  1. Erin Callihan, Nick Jensen, Astrid S Rodriguez, Andrina Musser, Bernard Savarese, Christine Facto-Reynoso, Holly Halmo, Kathleen Reilly, Phillip Hughes, Sofeia Eddy, hg47@nyu.edu

TRANSCRIPT EXAMPLE

ENTREPRENEURSHIP TRANSCRIPT

How to Read BobChat Transcript

Black text is what the user will see.

Blue and green text are back-end only.

For reference:

Within the chatbot, these will look like this:

Character limitations

Menu options: 20 characters

Gallery descriptions: 80 characters

Galleries are in tables like this

Descriptions go in this middle row

Buttons go on the bottom row

To access Entrepreneurship Section

        Say “hi” → Click “Development” → Click “Entrepreneurship”

        OR Type “Entrepreneurship”

Or Use this link: m.me/NYUBobChat?ref=Entrepreneurship

Entrepreneurship Transcript

Entrepreneurship Welcome

NYU fosters a University-wide startup ecosystem that encourages multi-disciplinary collaboration, accelerates innovation, and cultivates the next generation of startup leaders.

Entrepreneurship Contact

There are many ways to get in contact with them:

Entrepreneurship Address

The address for the Leslie eLab is:

16 Washington Pl.

New York City, NY 10003

Entrepreneurship Email

Their email address is entrepreneur@nyu.edu

Entrepreneurship Funding

Innovation Venture Fund

Innovation Corps (I-Corps)

Prototyping Fund

VC Pitchfest

VentureWell E-Team Grants

Want to find out more?

Submit a venture profile to be considered for funding

Develop scientific and engineering discoveries into commercial ventures

Receive funding for hardware or software prototyping to bring your ideas to life

Present to a panel of VC and angel investors and fellow NYU entrepreneurs

Get funds and learn how to turn your social impact ideas into startup ventures

More Info

More Info

More Info

More Info

More Info

Click here!

Entrepreneurship Spaces

#NYUFemaleFounders

LaGuardia Co-op

LaGuardia Studio

Leslie eLab

NYU Entrepreneurs Network

Want to find out more?

Fun and engaging events for women entrepreneurs

Collaborative technology working space that offering services and programming

State-of-the-art facility and advanced technology resource for the NYU community

6,800sqft. facility for entrepreneurs to connect, collaborate, and get resources

Organizations with a mutual goal of promoting and supporting entrepreneurship

More Info

More Info

More Info

More Info

More Info

Click here!

Entrepreneurship Workshops

Entrepreneurship & Innovation Courses

Lean Launchpad Educators Program

Master’s of Science in Computing, Entrepreneurship & Innovation (MS-CEI)

Innovation Corps (I-Corps) Summer Program

Startup School

Want to find out more?

Take courses around various schools to boost your entrepreneurial knowledge

Receive training on how to integrate Lean Startup principles into your teaching

Learn about the program combining Courant CS and Stern business courses

Develop scientific and engineering discoveries into commercial ventures

Get the training and resources to successfully launch and grow a startup

More Info

More Info

More Info

More Info

More Info

Click here!

Entrepreneurship Events

For a list of all upcoming events

INTERNSHIP DESCRIPTION

ABOUT US

The NYU Office of Student UX, Technology, and Engagement leverages research, data, and technology to craft meaningful student experiences.

Our team of innovators and strategists is driven by our fierce curiosity to learn from and with students to better understand their experience. We imagine solutions to enhance the NYU student journey, and we partner with university stakeholders to serve students with empathy and respect.

The office was originally formed in 2012 as the Office of Interactive Media, which earned national and international recognition from the American Marketing Association, HighEdWeb, NASPA, and others, for its innovative approach and collaborations.

STUDENT UX STRATEGY INTERNSHIP

We are seeking interns to help us improve the NYU experience by exploring digital innovations and providing support to their fellow students. Among other projects, interns will help manage NYU BobChat — a chatbot eager to help students find answers to questions, collect and send feedback, and navigate all things NYU.

Interns meet weekly with supervisors to discuss performance and share feedback, and evaluations take place at each semester's conclusion. While our internships are generally semester-by-semester, interns who prove to be a good fit will have the opportunity to continue on in the team.

RESPONSIBILITIES

OUR IDEAL INTERN

PLATFORMS GUIDE

COMMUNICATIONS & TASKS

SLACK

Slack is our default method of communication on a daily basis. We use two separate workspaces for different purposes. You must be logged into both during your office hours.

NYU OUX

All office communications, including announcements, resources, and conversations between team members.

NYU BOBCHAT

Conducting live chats and editing AI through Janis (more on these later). Do not use to communicate with team members.

AIRTABLE

Airtable is our task management and reference platform. We use three bases:

BOBCHAT

Tasks, updates calendar, live chat records, and "voicemails" and feedback submitted by users.

BOBCHAT CATALOG

A list of all BobChat content, used to search it more easily and keep track of which content is connected to the AI. This is a single-login base, shared by the whole team.

BOBCHAT LIVE CHAT REQUESTS

A list of all users who trigger our live chat feature, used for automation. You won't usually need to look into it.

CONTENT

We build BobChat's content — everything the user sees, from text to images to buttons — on Chatfuel. The platform is built around blocks and cards, grouped by subject or function.

CHATFUEL BASICS

BLOCKS & CARDS

Blocks are the basic components sent to users once they interact with the bot — like pages on a website. Cards are the individual points of information within each block — like each paragraph or image on a web page. Cards can contain text, buttons, images, redirects, etc. This asset guide lists image ratios, file types, character limits, and other helpful information when building blocks.

GROUPS

We organize blocks into several groups based on topic or function. You can find a full list and description of each group in the BobChat Catalog Airtable.

TABS

Chatfuel's dashboard is divided into the following tabs:

FORMATTING

?BLOCKS

We limit the amount of content listed in menus so that users are not overwhelmed. Any block not reachable through a menu is marked with a question mark. Users can only reach these blocks via the AI, so the blocks must be connected on Dialogflow.

"BACK BUTTON" QUICK REPLIES

We use three types of QRs that act as back buttons for users:

VIDEO CARDS
GALLERY CARDS
BUTTONS

To ensure uniformity, we use five phrases for buttons. Most are action phrases, so only the first word is capitalized.

ARTIFICIAL INTELLIGENCE

Dialogflow is where BobChat gets smart. Using Google's machine learning and natural language processing, Dialogflow creates "intents" containing examples of keywords that users may input, and matches those intents to the appropriate content on BobChat.

Janis.ai intermediates between Chatfuel, Dialogflow, and Slack. It allows us to easily create and edit Dialogflow intents and user inputs, and provides us a quick way to test BobChat's AI.

Janis also powers our user engagement, as we will detail later in the guide.

CREATING NEW INTENTS

You can create new intents either on Dialogflow or Janis. We recommend Janis since it's harder to make a mistake. Once you know which block you want to create an intent for:

ADDING INPUTS

Once an intent is created, it's time to add user inputs so that the block is easily accessible. Some general tips:

To add user inputs:

EDITING INTENTS

To delete intents or user inputs:

To edit an intent name (in case its Chatfuel block has been renamed):

To edit an intent's priority, making it easier or harder to trigger:

This feature is reserved for blocks that keep getting triggered when they aren't relevant at all. If we can't figure out which user inputs are causing the problem, we set the priority to low.

AUTOMATION

Integromat helps us automate vital tasks on BobChat via "scenarios." Most scenarios follow a similar pattern:

LIVE CHAT

Adds a new row to the BobChat Live Chat Requests Airtable base every time a user triggers a live chat. The row will generate a timestamp to determine if the request is within our live chat hours or not. Chatfuel will send either the "Open" or "Closed" blocks so that the user will know whether a live chat is starting soon, or whether to leave a message.

VOICEMAILS

Adds a new row to the BobChat Voicemails Airtable every time a user leaves a message outside of live chat hours. Also sends a message to #voicemails channel on the NYU BobChat Slack workspace.

FEEDBACK

Collects user data from feedback form and adds new rows to the BobChat Airtable.

In order to ensure we get as much data as possible even if the user stops the form halfway through, we added multiple JSON blocks. When users do complete the entire form, there will be duplicate data. You can delete the row(s) with the least amount of information once you've confirmed that they are from the same submission.

CONVERSATION RECORDS

Converts text records from the Engagement tab into Google Docs in the Conversation Records folder once a month. Since this is not connected to Chatfuel or JSON, it does not have a webhook.

BUILDING CODES

Searches our Building Codes directory and returns the address. If it can't find the code, it returns an error and allows the user to try again.

ENGAGEMENT GUIDE

INTRODUCTION

A key part of BobChat’s mission is to serve as a virtual concierge desk for users and students. If BobChat doesn’t know the answer to a question, users can request live chats with our team. The following is our basic approach for user interactions.

INTRODUCE YOURSELF

Tell the user your name and that you are not a bot. Any time that you are talking with a user, whether or not they requested a live chat, you must announce that you are a human.

SHOW EMPATHY

ISOLATE THE ISSUE

PROVIDE IMMEDIATE RELIEF

Find solutions or alternatives to help alleviate the problem at hand.

If you need time, let the user know — don't leave them hanging.

PROVIDE LONG-TERM STEPS FOR IMPROVEMENT

Inform the user on any resources available to address the underlying issue.

These are guidelines, not a hard script — feel free to tweak depending on the situation:

INTERVENTIONS

BobChat is a work in progress and will sometimes glitch or fail to provide an answer. When that happens, our policy is to reach out to users proactively to ensure they are not misinformed. Here are some example scenarios:

MISDIRECTS

User: When is Bobst open?

BobChat: The hours for Kimmel are [...].

You: Hi, I'm Nick — a human who works with BobChat. Looks like BobChat had a screw loose: Bobst is open [...]

"INCORRECT" ANSWER

User: When is Welcome Week this year?

BobChat: NYU Welcome Week
        When: Week before Fall Classes

        Over 500+ events for new undergraduate students

User: But what are the exact dates for Welcome Week this year?

BobChat: NYU Welcome Week
        When: [...]

You: Hi, I'm Nick — a human who works with BobChat. I see BobChat is giving you some trouble finding the answer you need. Welcome Week will be between [...]

MORE DETAIL NEEDED

User: Can you help me locate my advisor's contact information?

BobChat: Yikes! I couldn't find that. Want me to connect you with one of my humans?

You: Hi, I'm Nick — a human who works with BobChat. I see you're trying to find your advisor's contact but BobChat does not have an answer for you. If you'd like me to help you, please type "human" so we can have a live chat!

TROUBLESHOOTING

NOT KNOWING THE ANSWER

If a user asks a question during a live chat and you can’t find the answer for, just be honest.

Contact Tina or Nick for next steps.

PROTECTING PRIVACY

We do not collect private information from users, including N-Numbers, addresses, credit card numbers, Social Security numbers, and immigration statuses.

If an office requires a student's N-Number to solve an issue, refer the user to the office directly and explain why.

If a user volunteers sensitive information, remind them not to do so before continuing to assist them.

CHECKING ON EMERGENCIES

If a user indicates that they may be a danger to themselves or others, BobChat's emergency response will kick in automatically and Tina and Nick will be notified. The user will be sent contacts to 911 and NYU wellness resources. During on-call hours, we also reach out to users:

If the user is in genuine emergency (rather than testing the bot's response), urge them to contact 911 or the NYU Wellness Exchange. Collect as much information as you can from their Facebook profile and send to Tina and Nick immediately.

WORKFLOW

We use a variety of platforms and workflows to make live chats as smooth and helpful as possible for our users. Here are your tasks during on-call hours:

MONITORING

You are expected to monitor conversations and be ready to help users. To ensure a quick response, keep these platforms open at all times:

ENGAGING WITH USERS

Janis is an app connected to our Slack workspace that helps us conduct live chats. When a user receives an error message or requests human help, Janis notifies us. To chat with a user:

Ideally, interventions should happen on Janis so that you can pause or resume the bot, or send users back to the main menu if needed. However, when conversations aren't on Janis — because the user did not receive an error message or did not request human help — you can reply directly on Facebook Inbox. In this case, send only one message, either with the full answer or offering more assistance if they request a live chat.

PUSHING CONTENT

Janis allows us to send blocks directly to users with the "push content" command. This is useful in sending users back to the main menu at the end of every live chat, or by correcting a simple misdirect when BobChat already has the answer needed (bypassing an intervention).

Alternatively, if you already know the block title, you can send it in one step by typing "@janis push [Block Title]" (without the brackets). For example, the main menu block would be:

The command is case-sensitive, so you must type the block title exactly as it is on Chatfuel. You can find a list of all pushable blocks on our Bobchat Catalog. It's imperative that you have a good understanding of BobChat's content so you can push blocks as quickly as possible.

SUBMITTING ENGAGEMENT FORMS

After every live chat and intervention (but not content pushes that bypass interventions), you must submit an Engagement Form. You can find the form on our Knowledge Base or through Airtable. To complete it:

STYLE GUIDE

INTRODUCTION

When writing on behalf of BobChat, these are the guidelines you must follow. When speaking to users in your own voice during live chats, you are encouraged to keep them in mind — but feel free to use your own personal style.

GENERAL PRACTICES

VOICE AND TONE

FORMATTING

DATE & TIME

PUNCTUATION

ADDRESSES & CONTACT

ABBREVIATIONS AND SPELLINGS