Scoring Survey

JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

Scoring Survey

Thank you for answering a few questions about forecast scoring! Your responses here are incredibly important to us and will help us strengthen our scoring system.

Q0.1: The goal of this survey is to better understand user preferences and intuitions about scoring. We will use the results to help us improve scoring on Metaculus to make it as understandable, consistent, and intuitive as possible, while ensuring that we use rigorous and proper scoring methods.

There are no right or wrong answers for most of these questions. Many trade-offs ultimately come down to subjective preferences. Your answers to this survey will help us tremendously. Thank you in advance!

Please answer without doing any math or thinking deeply about tradeoffs. Don’t hesitate to answer “indifferent” or “I don’t know”. Those are valid, valuable answers!

We would like to get feedback from a wide range of forecasters from beginner to expert. We've attempted to order the questions from those that require no forecasting experience to more detailed expertise.

All questions are also skippable, though note we will probably interpret that as “indifferent” or “I don’t know.”

I understand

Required

Q0.2: What is your Metaculus username? (optional)

Q0.3: How long have you been using Metaculus?

I've never used Metaculus

less than a month

1 - 3 months

3 - 6 months

6 months - 1 year

1 - 2 years

2+ years

Clear selection

Q0.4: Do you know what a Brier score is? A log score? A proper scoring rule?

I’ve never heard of it

I’ve heard of it

I understand the general principle

I can write down the math without looking it up

Brier score

Log score

Proper scoring rule

Clear selection

Q0.5: How important is it to you to understand the math behind forecast scoring?

Not at all

Very much

Clear selection

Q0.6: Would you prefer that the range of scores you can get on a question was...

... between -10 and 1

... between -1000 and 100

... between 0 and 100

no preference

Other:

Clear selection

Q0.7: How much would you prefer the Metaculus scores to be based on standard and easy to understand and reproduce mathematics, vs. being less simple and custom-made for the platform?

I prefer simpler

I prefer custom

Clear selection

Q0.8: Please share any extra thoughts you may have about this section, and any of the above questions.

Clear form

Never submit passwords through Google Forms.

This form was created inside of metaculus.com.

Does this form look suspicious? Report

Forms