A Study of URL Scope in Web Applications
2. Info Whatsap And Firefox OS
Regarding the previous proposal of using the start_url or Document URL as a default origin:
Regarding the proposal to set an unbounded scope for apps by default:
Regarding the proposal to use the root of the web app's origin as the default scope:
The following are notes on the URL structures of the top 20 web sites in the Alexa Top 500 (as used in desktop Firefox), the Chrome Web Store (as used in Chrome OS) and the Firefox Marketplace (as used in Firefox OS).
The Alexa Top 500 is a list of top web sites on the web, but not all of these web sites would necessarily translate well into an “app” user experience. The web properties listed in the Chrome Web Store and Firefox Marketplace are more likely to have app-like characteristics because the developer has submitted them to an “app store”.
URLs considered “outside the app” are those which are linked to from the start URL but have their own visual identity and “feel” separate from the initial “app”. Indications might be that they have their own logo, their own navigation or their own style. This is obviously very subjective.
These tests were carried out in the United Kingdom and therefore some of the URLs automatically redirect to UK-specific versions.
Start URL
(redirects to https://www.google.co.uk/?gfe_rd=cr&ei=TrBjVK1T0qrzB9jagNAN&gws_rd=ssl)
Example URLs Inside App
Web search
https://www.google.co.uk/?gfe_rd=cr&ei=TrBjVK1T0qrzB9jagNAN&gws_rd=ssl#q=foo
Sign In
Example URLs Outside App
Image search
https://www.google.co.uk/imghp?hl=en&tab=wi&ei=TrBjVPW2CKiv7Abgy4D4DQ&ved=0CAQQqi4oAg
About
https://www.google.co.uk/intl/en/about/
Business Solutions
https://www.google.co.uk/services/?fg=1
Google Ads
https://mail.google.com/mail/u/0/?tab=wm&pli=1#inbox
Notes
Google search feels like it part of google.com, but Google Image search has its own logo and home page so might be considered a separate app like Mail, Calendar etc.
Start URL
Example URLs Inside App
Login
https://www.facebook.com/login.php?login_attempt=1
Wall
https://www.facebook.com/johnsmith
Events
https://www.facebook.com/events/upcoming
Messages
https://www.facebook.com/messages/john.smith
Photos
https://www.facebook.com/johnsmith/photos
Requests
https://www.facebook.com/friends/requests/?fcref=jwl
Notifications
https://www.facebook.com/notifications
Create Page
https://www.facebook.com/pages/create/?ref_type=logout_gear
Settings
https://www.facebook.com/settings
Games
https://www.facebook.com/games/?from_bookmark=1
Example URLs Outside App
Help
https://www.facebook.com/help/privacy/sharing-choices?ref=contextual
Start URL
https://www.youtube.com/
Example URLs Inside App
Sign In
My Channel
https://www.youtube.com/channel/UCW65Kls_xXp71iry-zsIXEg
Subscriptions
https://www.youtube.com/feed/subscriptions
History
https://www.youtube.com/feed/history
Channels
https://www.youtube.com/channels
Search
https://www.youtube.com/results?search_query=foo
Example URLs Outside App
About
https://www.youtube.com/yt/about/en-GB/
Press
https://www.youtube.com/yt/press/en-GB/
Copyright
https://www.youtube.com/yt/copyright/en-GB/
Terms
https://www.youtube.com/t/terms
Privacy
https://www.google.com/policies/privacy/
Start URL
(redirects to https://uk.yahoo.com/?p=us)
Example URLs Inside App
Sign In
https://login.yahoo.com/config/login?.src=fpctx&.intl=uk&.done=https%3A%2F%2Fuk.yahoo.com%2F
Example URLs Outside App
https://uk-mg42.mail.yahoo.com/neo/launch?.rand=9gqb23t3ef8q3
News
Finance
Celebrity
https://uk.celebrity.yahoo.com/
Search
https://uk.search.yahoo.com/search;_ylt=AnrE.vY4SZlNU0QkUnV50WM4hJp4?p=foo&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-903&fp=1
Notes
If you consider Yahoo Search (which has its own subdomain and homepage) as a separate app, and everything else like Mail, News, Finance, Weather (which also have their own home pages and visual identity) as separate apps, then there’s very little which is part of yahoo.com itself, that’s just a starting point to get into various apps.
Start URL
Example URLs Inside App
Search
Example URLs Outside App
News
http://news.baidu.com/ns?cl=2&rn=20&tn=news&word=ga&ie=utf-8&ie=utf-8
Music
http://music.baidu.com/search?fr=ps&key=ga&ie=utf-8&ie=utf-8
Maps
Notes
News, Music and Maps etc. could be considered separate apps, and each have their own subdomain. There is relatively consistent navigation across these various properties though.
Start URL
(user then presented a choice for their language http://en.wikipedia.org/wiki/Main_Page)
Example URLs Inside App
Article
http://en.wikipedia.org/wiki/Philae_%28spacecraft%29
Edit Article
http://en.wikipedia.org/w/index.php?title=Philae_%28spacecraft%29&action=edit
Talk Page
http://en.wikipedia.org/wiki/Talk:Philae_%28spacecraft%29
Help
http://en.wikipedia.org/wiki/Help:Contents
About
http://en.wikipedia.org/wiki/Wikipedia:About
Example URLs Outside App
WikiBooks
http://en.wikibooks.org/wiki/Main_Page
Terms of Use
http://wikimediafoundation.org/wiki/Terms_of_Use
Privacy Policy
http://wikimediafoundation.org/wiki/Privacy_policy
Donate
Notes
Wikipedia uses a subdomain per language and everything is fairly contained within that subdomain.
Start URL
(Redirects
Example URLs Inside App
Sign In
Search
http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=gda
Today’s Deals
http://www.amazon.com/gp/goldbox/ref=nav_cs_gb
Amazon Instant Video
http://www.amazon.com/Instant-Video/b/ref=nav_shopall_aiv?ie=UTF8&node=2858778011
Digital Music Store
http://www.amazon.com/MP3-Music-Download/b/ref=nav_shopall_digi_str?ie=UTF8&node=163856011
Help
https://www.amazon.com/gp/help/customer/display.html/ref=nav_cs_help?ie=UTF8&nodeId=508510
Sell
http://www.amazon.com/gp/seller-account/mm-product-page.html/ref=nav_cs_sell?ie=UTF8&ld=AZSOAUSCSNav
Example URLs Outside App
Careers
Investor Relations
http://phx.corporate-ir.net/phoenix.zhtml?c=97664&p=irol-irhome
Become an Affiliate
https://affiliate-program.amazon.com/gp/associates/join/landing/main.html
Amazon Fire TV
http://www.amazon.com/Streaming-Internet-Media-Player/oc/Fire-TV
Notes
Start URL
https://twitter.com/
Example URLs Inside App
Sign In
Forgotten Password
https://twitter.com/account/begin_password_reset?account_identifier=bfrancis
Signup
Profile
Notifications
https://twitter.com/i/notifications
Discover
https://twitter.com/i/discover
Search
https://twitter.com/search?q=foo&src=typd
Who to Follow
https://twitter.com/who_to_follow/suggestions
Example URLs Outside App
About
Jobs
https://about.twitter.com/careers
Blog
Developers
Privacy
Support
Notes
Start URL
(redirects to http://www.taobao.com/market/global/index_new.php)
Example URLs Inside App
Search
Product Detail
Example URLs Outside App
Alibaba Group
http://www.alibabagroup.com/cn/global/home
Notes
Very difficult to define the boundaries of this web property, there are multiple interlinked websites over a large number of origins. Most links open in a new tab.
Start URL
Example URLs Inside App
Example URLs Outside App
Notes
QQ is a portal similar to Yahoo which links to lots of other web properties spanning multiple origins. Many of these (e.g. mail.qq.com) would be considered “apps” in their own right and have their own homescreen and visual identity.
Start URL
https://www.google.co.in
Example URLs Inside App
Search
https://www.google.co.in/?gws_rd=ssl#q=ag
Image Search
https://www.google.co.in/imghp?hl=en&tab=wi&ei=DcJkVPOoEs73aoS5gsgK&ved=0CAQQqi4oAg
Example URLs Outside App
About
https://www.google.co.in/intl/en/about/
GMail
https://mail.google.com/mail/u/0/?tab=wm
Notes
Basically the same as google.com, but a version for India at its own origin.
Start URL
Example URLs Inside App
Login
Trouble signing in
Mail Inbox
https://snt152.mail.live.com/default.aspx
Sent Items
https://snt152.mail.live.com/?fid=flsent
Example URLs Outside App
Calendar
https://bay04.calendar.live.com/calendar/calendar.aspx
Word Online
https://office.live.com/start/word.aspx?auth=1&WT.mc_id=O16_Switcher_OneDrive_Outlook
Word Document
Notes
Used to be a search engine but now basically serves as a login page for the Outlook mail service.
Start URL
Example URLs Inside App
Forgotten Password
https://www.linkedin.com/uas/request-password-reset?trk=signin_fpwd
Logged in homepage
https://www.linkedin.com/nhome/?trk=
Messages
https://www.linkedin.com/inbox/#messages?trk=hb-messages-hdr-item-msg-v2
Invitations
https://www.linkedin.com/people/pymk/hub?ref=global-nav&trk=nav_utilities_invites_header
Profile
Advanced Search
https://www.linkedin.com/vsearch/f?adv=true&trk=federated_advs
Example URLs Outside App
About
http://www.linkedin.com/about-us?trk=hb_ft_about
Careers
http://www.linkedin.com/company/linkedin/careers?trk=hb_ft_work
Advertising
http://www.linkedin.com/advertising?src=en-all-el-li-hb_ft_ads&trk=hb_ft_ads
Notes
Start URL
Example URLs Inside App
Example URLs Outside App
About
http://corp.sina.com.cn/eng/sina_index_eng.htm
News
Notes
Mainly a portal to other content across multiple origins, probably doesn’t translate well into an app-like experience in its own right.
Start URL
Example URLs Inside App
Search
Register
http://register.tmall.com/?spm=a220m.1000858.a2226mz.3.lVL9LP
Login
Example URLs Outside App
Notes
A shopping portal which links into multiple sub-sites with their own origins. The registration page, login page and search results are also on separate origins.
Start URL
Example URLs Inside App
Signup
http://www.weibo.com/signup/mobile.php?lang=zh-tw&from=twweibo
World News
http://overseas.weibo.com/worldnews
Entertainment
Fashion
http://overseas.weibo.com/fashion
Search
http://s.weibo.com/weibo/foo
Example URLs Outside App
Help
Movies
Talk
Notes
More self-contained than other Chinese sites, mostly under one origin with lots of subdirectories, except for search which has its own origin. Links to sub-sites with their own origin and visual identity.
Start URL
(redirected from http://blogspot.com)
Example URLs Inside App
Profile
https://www.blogger.com/profile/122680544004560511
Login
https://accounts.google.com/ServiceLoginAuth
Edit Blog Post
https://www.blogger.com/blogger.g?blogID=265267023115423649#allposts/src=dashboard
Manage Comments
https://www.blogger.com/blogger.g?blogID=265267023115423649#publishedcomments
Example URLs Outside App
Blog
http://johnsmith.blogspot.co.uk
Help
https://support.google.com/blogger/#topic=3339243
Terms
http://www.google.com/intl/en/policies/terms/
Privacy
http://www.google.com/policies/privacy/?hl=en
Content Policy
https://www.blogger.com/content.g?hl=en
Notes
Blogging software. The blog editor behaves a lot like an app, the blog itself is on its own origin.
Start URL
Example URLs Inside App
News article
http://news.yahoo.co.jp/pickup/6138453
Example URLs Outside App
Docs
Notes
Mainly a portal to other web sites and web apps, like yahoo.com.
Start URL
Example URLs Inside App
TV
Movies
News
Example URLs Outside App
Search
http://www.baidu.com/s?word=foo&tn=sitehao123&ie=utf-8
Topic
News Article
http://news.china.com/focus/ddz/11163081/20141113/18960631.html?hao123
Notes
Many of the sub-sections of this site are loaded via AJAX and do not change the URL of the top level browsing context. Largely a portal to other web sites.
Start URL
Example URLs Inside App
Search
http://yandex.ru/yandsearch?lr=105001&text=foo&csg=482%2C1288%2C3%2C3%2C0%2C0%2C0
Example URLs Outside App
Maps
http://maps.yandex.ru/?ll=-1.613333%2C54.977778&spn=0.161362%2C0.081653&z=13&l=map
Market
http://market.yandex.ru/?clid=505
Notes
Search results on same origin, lots of links to other sub-sites at their own subdomains.r
Note: The “most popular” apps seem to be randomised so you’ll never see the same ten apps. This is just a selection. Packaged apps were excluded as by definition the scope of a packaged app is anything inside the package.
Start URL
Example URLs Inside App
Login
https://instagram.com/accounts/login/
Forgotten password
https://instagram.com/accounts/password/reset/
Example URLs Outside App
About
http://instagram.com/about/us/
Support
Blog
Notes
Start URL
https://www.evernote.com/Home.action
Example URLs Inside App
Registration
Login
https://www.evernote.com/Login.action
Main Page
https://www.evernote.com/Home.action
Settings
https://www.evernote.com/Settings.action
Example URLs Outside App
Products
https://evernote.com/products/
About
Notes
If you install Evernote as an app and set it to open in its own window then the settings screen acually opens separately from the app in the browser.
Start URL
https://office.live.com/start/word.aspx?WT.mc_id=016_Chrome_Web_Store_App_Word_1
Example URLs Inside App
New Document
Example URLs Outside App
Document list
https://onedrive.live.com/?id=2470E99F46C9BA37!106&cid=2470e99f46c9ba37
Notes
Start URL
Example URLs Inside App
Recent
https://onedrive.live.com/?qt=mru
Shared
https://onedrive.live.com/?qt=sharedby
Options
https://onedrive.live.com/options?ru=https%3a%2f%2fonedrive.live.com%2f
Example URLs Outside App
Help
http://windows.microsoft.com/en-gb/onedrive/onedrive-help#onedrive=other
Account Settings
Terms
http://windows.microsoft.com/en-us/windows/microsoft-services-agreement
Notes
Start URL
Example URLs Inside App
Apps
https://play.google.com/store/apps
Music
https://play.google.com/store/music
Movies
https://play.google.com/store/movies
Wishlist
https://play.google.com/wishlist
Example URLs Outside App
Buy a gift card
https://play.google.com/intl/en-GB_uk/about/giftcards/
Notes
Start URL
Example URLs Inside App
Example URLs Outside App
FAQ
http://www.rovio.com/en/support/faq&support_category_id=36
Notes
A single page app.
Start URL
https://play.spotify.com/
Example URLs Inside App
Login
https://www.spotify.com/uk/login/?forward_url=%2Fuk%2Faccount%2Foverview%2F
Browse
https://play.spotify.com/browse
Radio
https://play.spotify.com/radio
Your Music
https://play.spotify.com/collection/playlists
Follow
https://play.spotify.com/follow
Settings
https://play.spotify.com/settings
User Profile
Example URLs Outside App
Forgotten Password
https://www.spotify.com/uk/password-reset/?_ga=1.248610254.8358110.1415732233
Legal
Notes
Start URL
https://www.google.com/finance
Example URLs Inside App
News
https://www.google.com/finance/market_news?ei=z_1kVLngO8zFwAODh4DgDQ
Portfolios
https://www.google.com/finance/portfolio?action=view&pid=1&ei=CP5kVPmEG-aLwAP69IHYDQ
Example URLs Outside App
Help
https://support.google.com/finance/?hl=en&ei=CP5kVIHJO4fDwAPUhIGoDg
Notes
Start URL
Example URLs Inside App
Cheats
http://littlealchemy.com/cheats/
Example URLs Outside App
Notes
Start URL
Example URLs Inside App
Technology
Sports
Example URLs Outside App
News Story
http://www.bbc.co.uk/news/technology-30036057
Notes
Note: Many of the top 20 are packaged apps and therefore scope isn’t particularly relevant. By definition the scope of a packaged app is everything inside the package.
Packaged app.
Packaged app.
Packaged app.
Packaged app.
It seems there are a couple of directions this could be taken in. One is to try to accommodate all existing web content with a complex, flexible solution. The other is to go for a simpler solution which is not backwards compatible with some of the web.
A complex solution might be an unbounded scope by default and a manifest which looks like:
{
"start_url”: “http://foo.com/",
"scope": {
"include": ["/foo", "/bar"],
"exclude": ["/baz", "/qux"]
},
"stay_in_app": ["http://norf.com"]
}
The scope has both a whitelist and a blacklist and there's an additional list of URLs which are not part of the app but should stay in the context of the app (e.g. third party authentication). This is very flexible but I think the danger is that the matching rules start to get quite complex for the user agent to carry out on every top level navigation and we risk creating a complex declarative list like AppCache which is unwieldy to maintain.
At the other extreme, the default scope is the root of the domain and the manifest looks like:
{
"start_url": "/foo",
"scope": "/foo"
}
Scope is a single URL prefix. This would be simple for new web apps to use, but may provide a bad experience for a lot of existing content where web sites and web apps share URL spaces in complex ways. For example https://evernote.com/products/ would be forced to open inside the Evernote app, and users of GMail might be kicked out of the app into the browser when they want to log in.
If we were starting with a clean slate then my preference would be that the default scope is the root of the origin and the scope of the app is defined by a single URL prefix (or perhaps a whitelist array).
Unfortunately if we turned this feature on overnight in Firefox OS and Chrome OS for existing apps it would create a bad user experience for many existing apps which were not built to this pattern. In fact in the case of Firefox OS which has separate data jars for each app it might actually break apps which navigate to a separate origin in the top level browsing context for authentication purposes. It would also create friction for a lot of big name web properties to adopt Web Manifest because it would either require them to add in workarounds in their apps or to make big infrastructure changes in order to use a manifest.
So, should Web Manifest be backwards compatible with all web content, should it be a clean slate for new web content built with a clean URL scope, or should it take a middle ground?
An example of a middle ground might be:
{
"start_url": "http://foo.com/bar",
"scope": ["/bar", "/baz"],
"stay_in_app": ["http://qux.com"]
}
which incidentally is basically what Jonas proposed in https://groups.google.com/forum/#!topic/mozilla.dev.webapi/yqLffXgokuI
I think we could also make the root of the origin the default scope as in Jonas' proposal above, but to do this for existing apps we would need to give developers advance warning.
Below are some example manifests for the Alexa top ten web sites with the middle ground example given above.
{
"name": "Google Search",
"start_url": "https://google.com/webhp",
"scope": : ["/webhp"],
"stay_in_app": ["https://accounts.google.com"]
}
Note: I have assumed that a Google app would just be for Google Search.
{
"name": "Facebook",
"start_url": "https://www.facebook.com"
"scope": "/"
}
Notes: http://facebook.com/help would open in the app, but has a link back to the homepage.
{
"name": "YouTube",
"start_url": "https://www.youtube.com/"
"scope": ["/feed", "/channel", "/channels", "/results", "/playlist",
"/purchases", "/subscription_manager", "/upload", "/user"],
"stay_in_app": ["https://accounts.google.com"]
}
Notes: I may have missed some of the scope.
{
"name": "Yahoo Search",
"start_url": "https://uk.search.yahoo.com/"
"scope": ["/"],
"stay_in_app": ["https://login.yahoo.com"]
}
Note: I have used Yahoo Search as an example, yahoo.com itself wouldn't make a great app.
{
"name": "Baidu Search",
"start_url": "http://www.baidu.com",
"scope": ["/"],
"stay_in_app": "https://passport.baidu.com"
}
{
"name": "Wikipedia",
"start_url": "http://en.wikipedia.org/",
"scope": "/"
}
Note: Wikipedia would require one app per language as they use a separate origin for each language.
{
"name": "Amazon",
"start_url": "https://www.amazon.com",
"scope": ["/"],
"stay_in_app": "http://www.amazon.com"
}
Note: Amazon tends to switch between https:// and http:// URLs a lot which might cause problems for capturing scope.
{
"name": "Twitter",
"start_url": "https://twitter.com/"
"scope": "/"
}
Note: https://twitter.com/privacy would open in the app, but it has a link back to the homescreen
{
"name": "Taobao Search",
"start_url": "http://s.taobao.com/"
"scope": "/",
"stay_in_app": "https://login.taobao.com"
}
Note: Using Taobao Search as an example, most of the results would open in the browser. It would be very hard to create an app spanning the whole of Taobao.
{
"name": "新闻",
"start_url"http://news.qq.com/"
"scope": ["/"],
}
Note: Using QQ News as an example, the whole of QQ would probably not make a great app.
More Examples:
{
"name": "Facebook",
"start_url": "https://www.facebook.com",
"scope": {
“include”: "/",
"exclude": "/help"
}
}
{
"name": "YouTube",
"start_url": "https://www.youtube.com/",
"scope": {
“include”: ["/"],
"exclude": ["/yt", "/t"]
}
"stay_in_app": ["http://accounts.google.com"]
}
{
"name": "Wikipedia",
"start_url": "http://en.wikipedia.org/",
"scope": ["/”]
}
{
"name": "Twitter",
"start_url": "https://twitter.com"
"scope": ["/"],
}
{
"name": "LinkedIn",
"start_url": "https://www.linkedin.com",
"scope": ["/"],
"exclude": ["/about-us", "/company", "/advertising"]
}
{
"name": "Blogger",
"start_url": "https://www.blogger.com",
"scope": ["/"],
"exclude": ["/content.g"],
"stay_in_app": ["http://accounts.google.com"]
}
{
name: “Google”,
start_url: “http://google.com/webhp”,
scope: : [“/webhp”],
open_in_app: [“http://accounts.google.com”]
}
Notes from Firefox Marketplace
1. ConnectA2 (packaged app)
2. Info Whatsap And Firefox OS (packaged app)
3. MessageMe - Gives a launch_path of /index.html but redirects to / which would exit the app. The two main screens of the app are /#conversations and /#contacts which would also open outside the app.
4. SoundCloud - Doesn't provide a launch path, what would we use as scope?
5. PenguinPop - launch_path of / should be fine
6. PacMan - launch_path of / should be fine
7. YOUZEEK - launch_path of /helixmini.v4.aspx?source=FirefoxMarketplace login page at /Auths/AltDefault.aspx would exit the app
8. Wallpaper Wonderful - launch_path of /firefox/ should be OK, single page app
9. Zombie Apocalypse - launch_path of /zombie-apocalypse/index.html?platform=kirk&p=mozilla should be OK, single page app
10. Slice Fruits (packaged app)
11. Blozzle - launch_path of /Blozzle.php?pid=6900 Probably OK as mainly a single page app. Links to other games would open outside the app and high scores board is on a separate origin, but that's probably OK. Note: Registering a high score requires submitting a form to another origin.
12. Rooftops - launch_path of / but redirects to another origin, not sure what we can do about that
13. Fruit Cut Ninja - launch_path of /fruit-cut-ninja/index.html?p=mozilla single page app probably OK
14. Pasjans/Solitaire - launch_path of / and single page app probably OK
15. 12 games in 1 - launch path of /mozilla/12in1/ probably OK
16. Air Combat - launch_path of / probably OK
17. Duck Shooting - launch_path of /DuckShooting.php?pid=6900 probably OK
18. Blocked Car - launch_path of /BlockedCar.php?pid=6900 probably OK
19. Finger Print (packaged app)
20. Chess - launch_path of /index.html single page app probably OK
Box - no launch path, what would we use?
Evernote - launch path /Home.action, settings panel of /Settings.action would open outside the app
Facebook - launch path of / probably OK
Twitter - doesn't specify a launch_path, what would we use as default scope?
Digg - don't provide a launch_path
Fire Tasks - /FireTasks/app.html# probably OK
YouTube - launch_path of / probably OK
SoundCloud - no launch path
Pinterest (packaged)
Jolicloud - no launch path
BrowserQuest (Packaged)