SurroundSound, Bugsnag and SumoLogic�Tools for Fixing Errors
{
"Brad Urani" : "Staff Engineer",
"Company" : " ",
}
If we can't tell errors from noise, we can't fix errors
Caution!
Don't
Over-Engineer
@BradUrani
@BradUrani
raise 'Oh no!'
@BradUrani
raise RuntimeError.new('Oh no!')
@BradUrani
raise RuntimeError.new('Oh no!')
=
raise 'Oh no!'
=
raise RuntimeError, 'Oh no!'
@BradUrani
Error vs. Exception
Exception
NoMemoryError
SignalException
StandardError
ArgumentError
IOError
EOFError
RuntimeError
ZeroDivisionError
SystemStackError
@BradUrani
@BradUrani
def StandardError < Exception
end
def ZeroDivisionError < StandardError
end
@BradUrani
begin� �rescue => e� �end
@BradUrani
begin� �rescue StandardError => e� �end
@BradUrani
Default for raise
RuntimeError
Default for rescue
StandardError
class RuntimeError < StandardError
end
@BradUrani
begin� �rescue Exception => e� �end
@BradUrani
SystemStackError
NoMemoryError
SignalException::Interrupt
ScriptError::SyntaxError
@BradUrani
class OutOfNachosError < StandardError
end
�
raise OutOfNachosError.new('PANIC NOW!')
@BradUrani
Active Record
StandardError
ActiveRecord::ActiveRecordError
ActiveRecord::RecordInvalidError
ActiveRecord::ConnectionNotEstablished
ActiveRecord::RecordNotFound
ActiveRecord::StatementInvalid
ActiveRecord::WrappedDatabaseException
ActiveRecord::RecordNotUnique
ActiveRecord::InvalidForeignKey
@BradUrani
Rails
@BradUrani
def create
@user = User.new(params[:user])
if @user.save
redirect_to @client
else
render "new"
end
end
end
@BradUrani
def create
@user = User.new(params[:user])
if @user.save
redirect_to @user
else
render "new"
end
rescue
flash[:error] = "Please customer contact support"
render "new"
end
@BradUrani
Don't rescue just because you can
(It's ok to crash in rare or unknown circumstances)
@BradUrani
Computers
Users
Developers
@BradUrani
Computers
Users
user_message
redirects�
Developers
message
metadata
reports
logs
�
Class hierarchy
HTTP Status Codes
�
@BradUrani
Goal
@BradUrani
def create
@user = User.new(params[:user])
if @user.save
redirect_to @user
else
render "new"
end
rescue
flash[:error] =" Please customer contact support"
render "new"
end
@BradUrani
DO NOT SWALLOW ERRORS!!
DO NOT SWALLOW ERRORS!!
DO NOT SWALLOW ERRORS!!
Reporting
@BradUrani
@BradUrani
@BradUrani
@BradUrani
Standard Features
@BradUrani
Power User Features
@BradUrani
def create
@user = User.new(params[:user])
if @user.save
redirect_to @user
else
render "new"
end
rescue
flash[:error] =" Please contact customer support"
render "new"
end
@BradUrani
def create
...
rescue => e
flash[:error] << " Please contact customer support"
Procore::ErrorHandler.handle(e)
render "new"
end
@BradUrani
class Procore::ErrorHandler
def self.handle(error)
Bugsnag.notify(error)
end
end
@BradUrani
class Procore::ErrorHandler
def self.handle(error, severity: :error)
Bugsnag.notify(error, severity: severity)
end
end
@BradUrani
def create
...
rescue => e
flash[:error] << " Please contact customer support"
Procore::ErrorHandler.handle(e, severity: :warn)
render "new"
end
@BradUrani
class Procore::ErrorHandler
def self.handle(error, severity: :error)
Bugsnag.notify(error, severity: severity)
end
end
@BradUrani
class Procore::ErrorHandler
def self.handle(error, severity: :error)
Bugsnag.notify(error, severity: severity)
log_error(error, severity)
End
def self.log_error(error, severity)
Rails.logger.send(severity, build_message(error))
end
end
@BradUrani
def create
...
rescue => e
flash[:error] << " Please contact customer support"
Procore::ErrorHandler.handle(e, severity: :warn, metadata: {
tool: :onboarding
})
render "new"
end
@BradUrani
@BradUrani
@BradUrani
Custom
Error
Classes
Deep Call Stacks
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
We know here
What message to display here
result, err := SomeFunction()�if err != nil {� // handle the error�}�
@BradUrani
case File.read "hello" do
{:ok, body} -> IO.puts "Success: #{body}"
{:error, reason} -> IO.puts "Error: #{reason}"
end
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
raise here
rescue here
A Custom Error Hierarchy
Store::Error
Store::PurchaseError
Store::InventoryError
Store::CouponError
Store::OutOfSeasonError
Store::InadequateInventory
def create
…
rescue PurchaseError => pe
Store::ErrorHandler.handle(e, { user_id: @user.id }, :warn)
flash[:notify] << pe.user_message
end
@BradUrani
def create
…
rescue InventoryError => ie
Store::Handler.handle(e, { user_id: @user.id }, :warn)
redirect_to shopping_cart_path(...)
rescue CouponError => ce
Store::Handler.handle(e, { user_id: @user.id }, :info)
redirect_to apply_coupon_path(...)
end
@BradUrani
def create
…
rescue PurchaseError => pe
Procore::ErrorHandler.handle(e, { user_id: @user.id }, :warn)
flash[:notify] << pe.user_message
end
@BradUrani
Don't expose Exception#message to the user
@BradUrani
Store::Error
Store::SecurityError
Store::PermissionError
Store::UnauthorizedError
Store::ReadOnlyError
Store::InsufficientPermissionError
@BradUrani
class PurchaseController < ApplicationController
rescue_from Store::UnauthorizedError do |error|
Store::Handler.handle(e)
respond_to do |format|
format.html { redirect_to unauthorized_path, status: :unauthorized }
format.json { render json: { error.user_message }, status: :unauthorized }
end
end
end
@BradUrani
def show
Nacho.find(1_000_000_000)
end
# raises ActiveRecord::RecordInvalid
@BradUrani
def create
Nacho.create!({ cheesiness: 1_000_000 })
end
# raises ActiveRecord::RecordInvalid
@BradUrani
@BradUrani
{
"status": 404,
"error": "Not Found"
}
@BradUrani
module ActionDispatch� class ExceptionWrapper� cattr_accessor :rescue_responses� @@rescue_responses = Hash.new(:internal_server_error)� @@rescue_responses.merge!(� 'ActionController::RoutingError' => :not_found,� 'AbstractController::ActionNotFound' => :not_found,� 'ActionController::MethodNotAllowed' => :method_not_allowed,� 'ActionController::UnknownHttpMethod' => :method_not_allowed,� 'ActionController::NotImplemented' => :not_implemented,� 'ActionController::UnknownFormat' => :not_acceptable,� 'ActionController::InvalidAuthenticityToken' => :unprocessable_entity,� 'ActionController::InvalidCrossOriginRequest' => :unprocessable_entity,� 'ActionDispatch::ParamsParser::ParseError' => :bad_request,� 'ActionController::BadRequest' => :bad_request,� 'ActionController::ParameterMissing' => :bad_request,� 'Rack::Utils::ParameterTypeError' => :bad_request,� 'Rack::Utils::InvalidParameterError' => :bad_request� )� ...� end�end
A Custom Error Hierarchy
Store::Error
Store::PurchaseError
Store::InventoryError
Store::CouponError
Store::OutOfSeasonError
Store::InadequateInventory
class Store::Error < StandardError
attr_accessor :metadata
attr_accessor :severity
def initialize(message, metadata = {}, severity = :error)
super(message)
@metadata = metadata
@severity = severity
end
def user_message
I18n.t('errors.unknown')
end
end
class Store::PurchaseError < Store::Error
end
@BradUrani
class Store::InventoryError < Store::PurchaseError
end
@BradUrani
class Store::InadequateInventoryError < Store::InventoryError
def initialize(product_name, requested_qty, available_qty)
super('Inadequate Inventory', {
product_name: product_name,
requested_qty: requested_qty,
available_qty: available_qty
})
end
def user_message
I18n.t('errors.inadequate_inventory_error', product_name)
end
end
def create
…
rescue PurchaseError => pe
Procore::ErrorHandler.handle(e, { user_id: @user.id }, :warn)
flash[:notify] << pe.user_message
end
@BradUrani
def create
…
rescue InventoryError => ie
Store::Handler.handle(e, { user_id: @user.id }, :warn)
redirect_to shopping_cart_path(...)
rescue CouponError => ce
Store::Handler.handle(e, { user_id: @user.id }, :info)
redirect_to apply_coupon_path(...)
end
@BradUrani
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
raise
rescue
product_name
requested_qty
available_qty
user_id
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
raise
rescue
product_name
requested_qty
available_qty
user_id
coupon_code
???
def remove_items_from_inventory
...
rescue StoreError => se
se.add_metadata(coupon_code: code)
raise
end
@BradUrani
Why Rescue Errors and Re-Raise?
@BradUrani
PurchaseController#create
Purchase.start
remove_items_from_inventory
apply_coupon_code
clear_cart
process_payment
@BradUrani
ActiveRecord::Base.transaction do
remove_items_from_inventory
apply_coupon_code
clear_cart
process_payment
end
end
@BradUrani
Rescue and Continue
def get_shipping_price
get_quote_from_ups
rescue Store::ShippingPriceUnavailable => rce
Store::Handler.handle(rce)
6.99
end
end
@BradUrani
Raise and forget
@BradUrani
Raise and rescue in controller
@BradUrani
Raise, rescue, continue
@BradUrani
Wrapping
Errors
PurchaseController#create
Purchase.start
remove_items_from_inventory
item.remove_from_inventory
warehouse.make_unpurchaseable
warehouse.mark_as_purchased
warehouse_item.subtract_qty
warehouse.check_available!
def check_availability!
result = RestClient.get(...)
...
rescue RestClient::RequestTimeout => rte
raise Store::InventoryUnknownError.new(product_name, request_qty)
end
@BradUrani
def check_availability!
result = RestClient.get(...)
...
rescue RestClient::RequestTimeout => rte
raise Store::InventoryUnknownError.new(product_name, request_qty)
end
begin
check_availability!
rescue => e
puts e.cause.class
end
> RestClient::RequestTimeout < RestClient::RequestFailed
Why Wrap
@BradUrani
@BradUrani
Extras
def check_availability!
result = RestClient.get(...)
...
rescue RestClient::RequestTimeout => rte
raise Store::InventoryUnknownError.new(id: 'A3DHJ56')
end
Unique IDs at raise
@BradUrani
def check_availability!
result = RestClient.get(...)
...
rescue RestClient::RequestTimeout => rte
raise Store::InventoryUnknownError.new({
url: 'http://docs.procore.com/unknown_inventory_error'
})
end
Docs
@BradUrani
For every error raised
Rescued an Error
Security Error?
Show 404 Page
404
Page not Found
404
User not
Authorized?
Redirect to Auth Request Page
302
Unauthorized
401
Unknown Error?
Error Page (reraise)
500
Unknown Error
500
Availability
Error?
Redirect to Status Page
503
Status Error
503
Everything Else
Validation Error?
Data Error?
Reload Form
(handle in controller)
400
Error Map
Response
400
JSON
HTML
HTML
JSON
HTML
JSON
HTML
JSON
HTML
JSON
notify devops
notify responsible team
on Bugsnag
record to availability log
Further Reading
http://ieftimov.com/how-rails-handles-status-codes
http://blog.honeybadger.io/a-beginner-s-guide-to-exceptions-in-ruby/
http://blog.honeybadger.io/ruby-rescue-elegant-trick-for-knowing-which-exceptions-to-catch/
http://www.monkeyandcrow.com/blog/reading_rails_handling_exceptions/
@BradUrani
Who am I?
@BradUrani
I tweet at:
Connect with me:
I work in Santa Barbara at:
@bradurani@somewhy.com