BenchmarkDP consultation on software evaluation in digital preservation
Welcome to the BenchmarkDP consultation on software evaluation in the digital preservation field !
With this consultation we intend to get insights into the current state of software evaluations in the digital preservation field. We are interested in the current practices of software evaluations, challenges faced, limitations and advantages they bring to the running businesses. Finally, we are interested in better community-driven evaluation initiatives and want to gauge community members’ willingness to participate in such initiatives.

Who are we ?
BenchmarkDP is a research project funded by the Vienna Science and Technology Fund (WWTF). It has started in November 2012 and is running until summer 2016. You can find more information at http://benchmark-dp.org

Why participate ?
By participating in this consultation, you can help us to get a better overview of the software evaluation practices and results in the digital preservation field. This will enable us to invest our limited resources in the place where they are most needed and will have the biggest impact.

Who should participate ?
This consultation is open to everybody who wants to participate. We are interested to cover categories that are related to software tools and evaluations of those tools. So if you are a practitioner working with software tools in a memory institution, or a software developer developing solutions for digital preservation problems, or a researcher in the digital preservation area doing research related to software tools, you are most welcome to join.

How long does it take ?
This is a short consultation. It shouldn’t take you more than 10 minutes to finish it.

Privacy
This consultation doesn’t collect any private data unless you decide to provide your contact details. The only data it collects are the answers to the questions. The collected data will be used for the research purposes and summarized results will be publicly available. The data is collected through Google Forms and after the consultation is closed it will be stored on a secure server.

This consultation is open until 30th of November 2015.

1) What organization do you work for ? *
2) What is your job responsibility ? *
Mark those that apply
Required
3) What is your education background ? *
Mark those that apply
Required
4) How important are software evaluations and evaluation results for your and the work of your organization ? *
not important at all
very important
5) How do you and/or your team evaluate software tools ?
Mark those that apply
6) How are you and your organization affected by the current software evaluation practices and results in digital preservation? *
Mark those that apply
Required
7) What are you using evaluation results for ?
Mark those that apply
8) When you have done experiments to evaluate software tools, how did you conduct them ?
Mark those that apply
9) Which publicly available results have you used or produced?
This is an optional field where you can provide author and the title or links to the documents containing used results.
Your answer
10) Various challenges in software evaluation can make it difficult to get reliable results. Please rank the challenges according to how important they are for you. *
For each challenge assign a rank from 1 to 7. 1 is the highest rank. Each number can be used only once.
1
2
3
4
5
6
7
Lack of properly specified evaluation requirements.
Lack of dedicated resources to conduct evaluations.
Lack of tools and systems to support evaluation.
Lack of collaboration in the community.
Lack of properly annotated datasets (ground truth).
Lack of published evaluation results.
Lack of knowledge for doing proper evaluation.
11) Considering your work, rank the listed tool categories according to the need of having rigorous evaluation. *
For each category assign a rank from 1 to 6. 1 is the highest rank. Each rank can be used only once.
1
2
3
4
5
6
Characterization
Validation
Identification
Web Harvesting
Migration
Emulation
12) Rank the following aspects of software quality according to the importance for your work. *
For each aspect assign a rank from 1 to 3. 1 is the highest rank. Each rank can be used only once.
1
2
3
Coverage: Does the tool perform all the operations that are required to for a certain scenario?
Correctness: Does the tool perform operations which it claims to perform in a correct way?
Resource efficiency: How efficiently does the tool perform what is meant to do?
13) If the digital preservation community were to establish an initiative for software evaluation, would you be able to get involved? How?
Mark those that apply
Here you can provide any other opinions or facts you consider important for this topic but were not covered by the above questions.
Your answer
If you are interested in this topic and would like to participate in possible future interviews you can provide your email address below.
Your answer
Submit
Never submit passwords through Google Forms.
This content is neither created nor endorsed by Google. Report Abuse - Terms of Service