Upgrade Club

“Lightning Talk”

Agenda

About upgrade club on the community site

Our practice and experiences at Southampton

Open discussion

Upgrade Club

• Started 2016 having been given 3 months to arrange an upgrade.

​

• 2016 thread had 193 replies

​

• 2017 thread had 349 replies

​

• 2018 thread has 56 replies so far

Upgrade Club

• Threads are full of useful tips, questions, groans of disappointment and exaltations of success

​

• Members have a wide variety of backgrounds and skills

​

• Very positive, open, collaborative environment

​

Questions

• Have you posted in Upgrade Club?

​

• Are you upgrading Blackboard this year?
• To what version? 2016 Q4, 2017 Q2, 2017 Q4, 2018 Q2?

​

• Are you self-hosted / managed hosted / SAAS?

​

​

Southampton Upgrade History

• Started with Blackboard 5.0 in 2000
• 16 Major upgrades since then.

​

• Did you know?
• You can review your upgrade history by going to
• System admin 🡺 System configuration 🡪 System Information

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Southampton Bb Upgrades

• Working in a large IT department
• Prince 2, ITIL, Lean Six Sigma
• Very hierarchical and structured
• If you can imagine it - there is a board you have to go to discuss it
• Upgrades done out of hours = overtime = requires money = requires more project documentation

​

​

IT dept organogram

​

​

• ~375 staff

​

• Bb upgrades involve staff from multiple teams

​

Bb upgrade projects at Southampton

Project elements

• Project Brief
• Business Case
• Monthly Highlight Reports and Directorate Review
• Project Tasks for almost every activity (about 50 tasks for an upgrade)
• Arrange downtime

Upgrade work

• Upgrades to new release in devs and preprod
• Testing
• Disaster Recovery
• Upgrades to latest CU
• Documentation (internal and end-user)

Go live

• Change Management Approval
• Communications
• Weekend Upgrade
• Monitoring

​

Closure

• Lessons learned
• End project report
• Benefits realization review
• Update service roadmap
• Prepare for next project

Questions

• Do you run Blackboard upgrades as projects / using project management methodology?

​

• Do you do Blackboard upgrades out of hours?

​

• Is it simple to arrange budget for overtime / TOIL?

​

​

Working on Bb upgrades: lessons, recommendations, experiences

• A post giving an overview is at https://bit.ly/2HaDt1c (from Upgrade Cohort 2018 community area)
• Upgrade club blog: https://bit.ly/2qgL7MY
• The next slides are some of my “highlights”

Nothing here is “special”

• Every institution is different

​

• Nothing in this presentation is out of the ordinary

​

• Make sure to see Jonathan Knight’s presentation “Early Adoption and Minimal Testing”

​

​

​

Building the business case

To get budget to pay for overtime, need a full project business case.

This includes such items as:

Reasons to upgrade, benefits, resource requirements and costs, risks, etc..

Business case for our 2018 Blackboard upgrade was 37 pages long

Business Case tips

• Reasons

​

​

​

​

​

​

​

​

​

• http://library.blackboard.com/docs/support/Blackboard_Learn_Support_Services_Guide.pdf

Business Case tips

• Benefits

​

​

​

​

​

​

​

​

​

• Screen grab roadmap webinars to use as evidence of benefits from new versions

Question

• Do you perform a benefits realisation analysis following upgrades?

​

​

Implementation Plan

Start building as soon as you can.

I find using a wiki very useful. It’s quick to edit and can be structured so that plans can be copied easily and elements edited

Having a great plan that can be updated each year will save time.

Structure of a typical upgrade plan

1. Prepare files (installer, installer properties, back up files that installer deletes)
2. Oracle patching
3. Upgrade Blackboard
4. Review configuration changes
5. Make configuration changes
6. Pushconfigupdates
7. GUI based config and testing
8. Removal of temporary files (installer, backed up files etc.)

Verification scripts

Installer will wipe your carefully set fixes, workarounds, optimizations.

This year we started building verification scripts to quickly identify whether settings needed to be reset

Repeat verification scripts after doing pushconfigupdates

Protip – use VMware Snapshots

• We take a VMware “cold snapshot” of our vApp after each upgrade stage.

​

​

• If something goes wrong we can restore environment back to how it was within 15 minutes.

Keep up to date with issues / recommendations

From the community

• Mailing lists still have the best info (e.g. ASU BB-ADMIN-L)
• Community site has lots of useful info and a good place to ask questions
• BB World / BB TLC / User groups / DevCon

​

From Blackboard

• Generate known issues lists from support site
• Subscribe to all support notifications
• Bb support will often give extra help
• Build a good relationship, complete support surveys

From within

• Make your plans open within your department
• Encourage feedback and ideas
• Share lessons, build an environment of collaboration

Bb notifications

Question

• Do you have any other tips on keeping up to date with known issues?

​

​

Upgrade frustrations

• Installer will overwrite fixes Blackboard support asked you to implement to resolve known issues.
• Some are fixes are more than 4 years old (e.g. 000039703, 000037634)
• So you have to implement them again.
• For settings within bb-config.properties you can set most of these in the installer.properties file.

Additional settings we are using in the installer.properties files

• bbconfig.jvm.options.extra.tomcat
• bbconfig.jvm.options.gc
• bbconfig.email.use.dmarc.from.override
• bbconfig.max.stacksize.tomcat
• bbconfig.appserver.http.compression
• bbconfig.jvm.options.codecache.reserved

​

• bbconfig.jvm.options.codecache.initial
• bbconfig.cs.database.maxpoolsize
• bbconfig.peer.discovery.timeout.inactive
• bbconfig.peer.discovery.timeout.dead
• bbconfig.server.backend.processor
• bbconfig.gradecenter.cache.grade_threshold

Document “fixes” separately

• Those key fixes and workarounds can get “lost” in implementation plans.
• I found some fixes we implemented in 2014 had been lost in our 2016 upgrade because no one was left who knew about them.
• Keeping a separate list of fixes that should be re-applied until they are resolved centrally should save time and ensure they are not “lost”.
• Ours is now 17 pages long (22 fixes)

My “favourite” fixes so far

Installer failed for no apparent reason. Cause: random number generator not random enough (Thanks to Cherif Abbes /Bb for the fix)

SCORM disconnection fix (fix was to update click jacking settings) (Thanks to Stuart Robinson and the team at Leeds for the fix)

High CPU / Load caused by MicrosoftDocumentParser.sh (Thanks to Chris Filkins for the fix)

More Upgrade frustrations

• Upgrades that remove functionality without replacing it
• E.g.
• Virtual classroom and chat
• Crocodoc functionality loss
• Upgrades that add functionality which is broken
• E.g. availability toggle in 2017 Q4

​

Question

• What are your upgrade frustrations?

​

​

Testing

• We test core functionality and integrations.
• We accept we can’t test everything and rely on
• Bb support notifications
• Mailing List / Community site
• Amy Eyre from York for tipping us off about new issues
• We also perform a disaster recovery exercise and a load testing exercise.

Communications during upgrade

Outage page with embedded twitter feed

Keep an updated ETA completion

Visual indicator of progress (pie chart)

Celebrating success

Celebratory fried breakfast paid out of project budget (but not allowed to do this any longer ☹)

Ensure overtime payments / TOIL processed quickly

Arrange “thank you” email from University executive

Summary

Prepare

While onerous, building methodical project documentation is helpful in the long-term and often a requirement for funding.

Research

Get on the mailing lists, subscribe to Bb notifications, use the community, contribute to upgrade club ☺. Be nice to Bb support!

Upgrade

Careful documentation and verification essential (before we do our live upgrade we will have practiced it six times already).

Celebrate and learn

Celebrate success and note lessons and recommendations for next time.

Questions and discussion

​

​