| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | W | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Title | Author | Year | Abstract | Citation | URL | Paper type | Thrust/Thesis | Details | Population | Relevant to RQ | Relevant to RQ | How Relevant to RQ | Contribution | Limitations | Takeaways | Topic List Category | Tags | Unique ID | ||
2 | On the collaborative practices of cyber threat intelligence analysts to develop and utilize tacit Threat and Defence Knowledge | Ahrend, Jan M.,"Jirotka, Marina","Jones, Kevin" | 2016 | While the need for empirical investigations of cybersecurity analysts’ collaborative work practices is widely acknowledged, research efforts are fairly limited. This paper aims to provide empirical evidence to support a deeper consideration for the seemingly intangible collaborative practices that situational awareness in cybersecurity relies on and add to our understanding of what it means to “do” threat intelligence. In particular, it aims to unpack the informal forms of collaboration and coordination at work that build tacit knowledge about threat actors and defenders and that span across time, people and tools to inform the translation of threat information into actionable threat intelligence. In-depth semi-structured interviews and diary studies are conducted at three cyber threat intelligence service providers (N=5) and analyzed using thematic analysis. This paper introduces the concept of Threat and Defence Knowledge, tacit knowledge that analysts within an organization form over time and utilize through informal ways of becoming aware of this knowledge, making it available and correlating it. We find that a lack of accessibility to knowledge about relevant threat and defence factors can reduce analysts’ effectiveness at arriving at actionable threat intelligence and hence reduce the ability to be alerted in advance about cyber threats, to contain damage and obtain situational awareness. Perceived and potential shortcomings of the existing processes and tools are presented, and practices to circumvent the existing systems investigated and implications for design are considered. | J. M. Ahrend, M. Jirotka, and K. Jones, On the collaborative practices of cyber threat intelligence analysts to develop and utilize tacit Threat and Defence Knowledge, in 2016 International Conference On Cyber Situational Awareness, Data Analytics And Assessment (CyberSA), Jun. 2016, pp. 1–10. doi: 10.1109/CyberSA.2016.7503279. | doi.org/10.1109/CyberSA.2016.7503279 | Theory/Model | Aims to explore threat analysts’ day-to-day activities/workflow and formal/informal collaboration practices | UK IC. Focuses on cybersecurity (as the domain) and threat intelligence (TI). Semi-structured interviews with 5 workers from “TI providers“ (assuming govt contractors), some analysts, some in other roles in the org. Interviews covered topics: analysts’ contextual perception of their role within their organization, their workflow, interactions with technology, collaboration with others | 5 Workers from UK “threat intelligence providers“ (assuming govt contractors) | RQ1-Info | TDK can be thought of as an information type. Participants also described other attributes of the information they deal with in interviews | Introduces concept “Threat and Defence Knowledge“ - tacit knowledge that analysts develop and utilize informally. | Authors state that TI work is highly collaborative, both formally and informally, but do not go in depth about nature of collaboration, who is involved in collaboration, how collaborative connections are formed, etc. | Human Study People - Methodologies | Interview Study | 6 | ||||
3 | A user-centered look at glyph-based security visualization | Komlodi, A.,"Rheingans, P.","Ayachit, Utkarsha","Goodall, J.R.","Joshi, Amit" | 2005 | This paper presents the intrusion detection toolkit (IDtk), an information visualization tool for intrusion detection (ID). IDtk was developed through a user-centered design process, in which we identified design guidelines to support ID users. ID analysts protect their networks by searching for evidence of attacks in ID system output, firewall and system logs, and other complex, textual data sources. Monitoring and analyzing these sources incurs a heavy cognitive load for analysts. The use of information visualization techniques offers a valuable addition to the toolkit of the ID analyst. Several visualization techniques for ID have been developed, but few usability or field studies have been completed to assess the needs of ID analysts and the usability and usefulness of these tools. We intended to fill this gap by applying a user-centered design process in the development and evaluation of IDtk, a 3D, glyph-based visualization tool that gives the user maximum flexibility in setting up how the visualization display represents ID data. The user can also customize whether the display is a simple, high-level overview to support monitoring, or a more complex 3D view allowing for viewing the data from multiple angles and thus supporting analysis and diagnosis. This flexibility was found crucial in our usability evaluation. In addition to describing the tool, we report the findings of our user evaluation and propose new guidelines for the design of information visualization tools for ID. | A. Komlodi, P. Rheingans, U. Ayachit, J. R. Goodall, and A. Joshi, A user-centered look at glyph-based security visualization, in IEEE Workshop on Visualization for Computer Security, 2005. (VizSEC 05)., Oct. 2005, pp. 21–28. doi: 10.1109/VIZSEC.2005.1532062. | doi.org/10.1109/VIZSEC.2005.1532062 | Application/Design Study | Focus group informs a Toolkit for intrusion detection | Conduct Focus group - Provided prototypes and asked to deliberate, ask questions, offer opinions and sketch improvements. Design tool User evaluation - Describe the data, and how you'd know about false alerts, Create a news data mapping that would be helpful, identifiy other helpful mappings by prior participants. | 7 SNORT Users (intrusion detection speciallists) | RQ1-Info | RQ4-Collab | Asked users to describ the kinds of data they work with and how they would like it visualized to make it usseful. | Creates a tool with two modes - one for monitorying and another for analysis of multidimentional relationships | Artifact interview may not allow for completely new designs. | Tools/Systems | 7 | ||||
4 | Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study | Kang, Youn-ah,"Gorg, Carsten","Stasko, John" | 2009 | Despite the growing number of systems providing visual analytic support for investigative analysis, few empirical studies of the potential benefits of such systems have been conducted, particularly controlled, comparative evaluations. Determining how such systems foster insight and sensemaking is important for their continued growth and study, however. Furthermore, studies that identify how people use such systems and why they benefit (or not) can help inform the design of new systems in this area. We conducted an evaluation of the visual analytics system Jigsaw employed in a small investigative sensemaking exercise, and we compared its use to three other more traditional methods of analysis. Sixteen participants performed a simulated intelligence analysis task under one of the four conditions. Experimental results suggest that Jigsaw assisted participants to analyze the data and identify an embedded threat. We describe different analysis strategies used by study participants and how computational support (or the lack thereof) influenced the strategies. We then illustrate several characteristics of the sensemaking process identified in the study and provide design implications for investigative analysis tools based thereon. We conclude with recommendations for metrics and techniques for evaluating other visual analytics investigative analysis tools. | Y. Kang, C. Gorg, and J. Stasko, Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study, in 2009 IEEE Symposium on Visual Analytics Science and Technology, Oct. 2009, pp. 139–146. doi: 10.1109/VAST.2009.5333878. | doi.org/10.1109/VAST.2009.5333878 | Technique | Primary goal: understand how visualization tools can assist investigative analysis (how do people approach analysis with visualization tools and what benefit to visualization tools provide?) Secondary goal: inform evaluation methodologies for investigative analysis systems | 16 non-analyst participants divided across 4 investigative methods, one of which being the tool Jigsaw which they evaluated, did a 90 minute simulation of an investigative analysis. | Graduate students | RQ2-Process | Identified 4 styles of analysis that participants engaged with. Identified 5 useful elements of the analysis tool evaluated. Methodological contribution for future evaluation of investigative tools (most useful to us) | Participants were not analysts (random graduate students). Identified analysis techniques may not be applicable to analysts, e.g., in one analysis technique, participants searched keywords like "terrorist" which is a much too broad approach for an analyst to take. Also, the investigative task had a correct answer, whereas other studies seem to be more effective in emulating intelligence activities when there is not a clear delineation between relevant/irrelevant information. | Human Study People - Methodologies | Observational Study | 8 | |||||
5 | Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study | Kang, Youn-ah,"Stasko, John" | 2011 | While intelligence analysis has been a primary target domain for visual analytics system development, relatively little user and task analysis has been conducted within this area. Our research community’s understanding of the work processes and practices of intelligence analysts is not deep enough to adequately address their needs. Without a better understanding of the analysts and their problems, we cannot build visual analytics systems that integrate well with their work processes and truly provide benefit to them. In order to close this knowledge gap, we conducted a longitudinal, observational field study of intelligence analysts in training within the intelligence program at Mercyhurst College. We observed three teams of analysts, each working on an intelligence problem for a ten-week period. Based upon study findings, we describe and characterize processes and methods of intelligence analysis that we observed, make clarifications regarding the processes and practices, and suggest design implications for visual analytics systems for intelligence analysis. | Y. Kang and J. Stasko, Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study, in 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), Oct. 2011, pp. 21–30. doi: 10.1109/VAST.2011.6102438. | doi.org/10.1109/VAST.2011.6102438 | Application/Design Study | Learn the process and challenges of young analysts | 14 analyst-in-training participants divided into 3 groups completed a 10 week long intelligence analysis simulation project | Students in analyst program | RQ2-Process | 11 design implications for an IA visual analytics tool (too much to list here, but presented in a bulleted list on the last page) | Study was a team project which necessitates collaboration and may not reflect accurate collaboration practices of analysts | Human Study People - Methodologies | Observational Study | 9 | |||||
6 | POLESTAR – Collaborative Knowledge Management and Sensemaking Tools for Intelligence Analysts | Nicholas J. Pioch, John O. Everett | 2006 | In this paper, we describe POLESTAR (POLicy Explanation using STories and ARguments), an integrated suite of knowledge management and collaboration tools for intelligence analysts. POLESTAR provides built-in support for analyst workflow, including collection of textual facts from source documents, structured argumentation, and automatic citation in analytic product documents. Underlying POLESTAR is a scalable dependency repository, which provides traceability from product documents to source snippets. The repository’s notification engine allows POLESTAR to alert analysts when dependent sources are discredited and aid them in repairing affected arguments. The paper then discusses recent extensions to POLESTAR to support collaborative analysis through community-of-interest finding, portfolio sharing, and peer review of arguments. We conclude with a preview of future research and summary of POLESTAR’s primary benefits from the point of view of its deployed users. | Pioch, N. J., & Everett, J. O. (2006, November). POLESTAR: collaborative knowledge management and sensemaking tools for intelligence analysts. In Proceedings of the 15th ACM international conference on Information and knowledge management (pp. 513-521). | https://dl.acm.org/doi/pdf/10.1145/1183614.1183688 | System | Proposes a system, or "analytic environment", that promotes formation of ad hoc collaborative teams that span multiple agencies and jurisdictions to enhance information finding. | System has two aspects, one about individual analysts' use and one for collaborative work. Individual - Snippet collection (allows annotating in Microsoft Word and browser), Portfolio browser (Point of storage for all collected facts, similar to windows file storing system), Knowledge Structuring (Wall of Facts where snippets of information can be dragged in for sensemaking, can sort temporally, Argument Tree Editor with a hypothesis -> Supporting Claim -> Supporting Facts -> Rebutting Claim structure). Extremely cool feature - Recalled Document Notification (Notify analysts pulling data from the same document if someone proves that document or its source is now unreliable) using dependency tracking Collaborative Sensemaking - Community-of-interest Formation (Dependency view expanded to include artifacts, a link connects two analysts if they have both collected snippets of information from the same document), Portfolio Sharing (basically a shared space to let anyone in that community edit), Peer Review (An analyst invites desired analysts to review their report) | RQ2-Process | RQ4-Collab | A novel system supporting analyst workflows | Study from 2006 (so dated), no usability testing (that I could find), No justification or grounding for what they did or what features they implemented | An important system to look at in the sense that it implements some of the features that we've been talking about | Tool people - making data analysis tools and supporting intelligence | Tools/Systems | 10 | ||||
7 | Exploring the analytical processes of intelligence analysts | Chin, George,"Kuchar, Olga A.","Wolf, Katherine E." | 2009 | We present an observational case study in which we investigate and analyze the analytical processes of intelligence analysts. Participating analysts in the study carry out two scenarios where they organize and triage information, conduct intelligence analysis, report results, and collaborate with one another. Through a combination of scenario-based analysis, artifact analysis, role-playing, interviews, and participant observations, we explore the space and boundaries in which intelligence analysts work and operate. We also assess the implications of our findings on the use and application of key information technologies. | G. Chin, O. A. Kuchar, and K. E. Wolf, Exploring the analytical processes of intelligence analysts, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston MA USA, Apr. 2009, pp. 11–20. doi: 10.1145/1518701.1518704. | https://dl.acm.org/doi/10.1145/1518701.1518704 | Application/Design Study | Understand and explore the space and boundaries in which intelligence analysts work and operate using artifact interview and observations. Discuss the implications of our findings on use and application of info technologies. | 5 intel analysts participated in a two-scenario observation study. Scenario 1 - IAs conduct intelligence gathering and analysis individually 2 weeks to write an analysis report Asked to walk through analysis process at the end Scenario 2 - IAs collaborate in real-time as they jointly carry out intel analysis Static as well dynamic source information 1 hour to complete the task Results - Intelligence Analysis Strategies: Analysis of competing hypothesis had one in support and one against Ultimately, systematic approaches are often abandoned to fit time constraints Information collection and Triage: IAs more comfortable with what they've been using as tools/techniques (here even though all info was in electronic form, all 5 took physical prints of the documents) Identifying patterns: IAs look for concept details that are similar or that seem to co-exist in time, spcae, and other dimensions Personal knowledge plays a key role during this process IAs admit the fact that each agency wants to be the first to identify and eliminate threats, so collaboration is ususally avoided. Implications for Info Technologies: 1. Auto-generate a set of standard analysis perspectives given a set of facts and relationships 2. Workflow management systems 3. Sketching tools 4. Link analysis tools 5. Case management tools for collecting, managing, and querying past actions | Real analysts | RQ2-Process | RQ4-Collab | Highly relevant as both methodologies and research questions are pertinent to our discussion. | Future research directions based on current IAs' needs | Only 5 IAs who had all worked with each other before. This could have affected how well they'd work in a collaborative environment. | Human Study People - Methodologies | Exploratory | 11 | |||
8 | Helping Intelligence Analysts Make Connections | M. Shahriar Hossain, Christopher Andrews, Naren Ramakrishnan, and Chris North | 2011 | Discovering latent connections between seemingly un- connected documents and constructing “stories” from scattered pieces of evidence are staple tasks in intelli- gence analysis. We have worked with government in- telligence analysts to understand the strategies they use to make connections. Beyond techniques like cluster- ing that aim to provide an initial broad summary of large document collections, an important goal of an- alysts in this domain is to assimilate and synthesize fine grained information from a smaller set of foraged documents. Further, analysts’ domain expertise is cru- cial because it provides rich contextual background for making connections and thus the goal of KDD is to augment human discovery capabilities, not supplant it. We describe a visual analytics system we have built— Analyst’s Workspace (AW)—that integrates browsing tools with a storytelling algorithm in a large screen display environment. AW helps analysts systematically construct stories of desired fidelity from document col- lections and helps marshall evidence as longer stories are constructed. | Hossain, M. S., Andrews, C., Ramakrishnan, N., & North, C. (2011, August). Helping intelligence analysts make connections. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence. | https://www.aaai.org/ocs/index.php/WS/AAAIW11/paper/view/3937/4323 | System | Understanding analysts' process and proposing a tool to support sensemaking | Authors conducted interviews with analysts to understand how they approached large quantities of data they were required to sift through, and what they tools they use in the process. They found that tools were essentially used only when data came in (processing) and went out (delivering a report). The actual sensemaking process did not involve use of tools. Rest of the paper talks about a tool (Analysts Workspace or AW) that uses a multi-screen setup. This tool aims to replace hard copies of documents used in analysis and instead provide more screen space to promote annotations and interactions of that nature. | Analysts | RQ2-Process | 12 | |||||||||
9 | How Analysts Think: How do Criminal Intelligence Analysts Recognise and Manage Significant Information? | Celeste Groenewald, B.L. William Wong, Simon Attfield, Peter Passmore, Neesha Kodagoda | 2017 | The Criminal Intelligence Analyst’s role is to create exhibits which are relevant, accurate and unbiased. Exhibits can be used as input to assist decision-making in intelligence-led policing. It may also be used as evidence in a court of law. The aim of this study was to determine how Criminal Intelligence Analysts recognise and manage significant information as a method to determine what is relevant for their attention and for the creation of exhibits. This in turn may provide guidance on how to design and incorporate loose and flexible argumentation schemas into sense-making software. The objective is to be informed on how to design software, which affords Criminal Intelligence Analysts with the ability to effortlessly determine the relevance of information, which subsequently could assist with the process of assessing and defending the quality of exhibits. | Groenewald, C., Wong, B. W., Attfield, S., Passmore, P., & Kodagoda, N. (2017, September). How analysts think: How do criminal intelligence analysts recognise and manage significant information?. In 2017 European Intelligence and Security Informatics Conference (EISIC) (pp. 47-53). IEEE. Chicago | https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=82407688 | Application/Design Study | Understand how analysts recognize significant information and then further manage it. Cognitive Task Analysis (CTA) interviews conducted with 11 analysts from the UK and Belgium. | RQ1: How do analysts recognize significant information? Three instances that prompt attention for consideration of significance - 1) certainty of info 2) interestingness 3) strangeness Comes from experience and the context of overall information RQ2: How do analysts manage significant information? Strategy called the 'Lifecycle of Entities': 1) Discover interesting information 2) Discover a pattern 3) Establish communication 4) Explore a hunch 5) Confirm identities | Analysts | RQ2-Process | Slightly relevant, talks in very abstract language which may be difficult to situate in our work. | Understanding and abstraction of an analyst's thinking process when faced with big data. | Interview Study | 13 | ||||||
10 | Improving information requirements determination: a cognitive perspective | "Browne, Glenn J.","Ramesh, V." | 2002 | Requirements determination is a critical phase of information systems development, but much evidence suggests that the process can and should be improved. Because the bulk of requirements determination occurs early in the development of a system, improvements can yield significant benefits for the entire systems development process. This paper first discusses a three-stage descriptive model of the requirements determination process. Four classes of difficulties in determining systems requirements are then used to organize and describe particular problems that occur within each stage of the process, together with the cognitive and behavioral theories that underlie them. The paper then describes techniques that can address the problems and presents theoretical considerations that analysts can use in applying the techniques to improve requirements determination. | G. J. Browne and V. Ramesh, Improving information requirements determination: a cognitive perspective, Information & Management, vol. 39, no. 8, pp. 625–645, Sep. 2002, doi: 10.1016/S0378-7206(02)00014-9. | https://www.sciencedirect.com/science/article/pii/S0378720602000149?via%3Dihub | Theory/Model | Used cognitive psychology to better understand the information requirements of users. Lays out a framework for extracting this information from user behaviors. | Describes the methods that determine Inforamtion requirments. [will add more if the team is interested] | Literature Review | RQ2-Process | Describes to the designer a handful of "Techniques to Improve Requirement Determiniation" when talking with your target users. | Identifies and recommends approaches to challenges in understanding user needs. 'Analyst' in this paper is in reference to people cunducting research on people | Methodology | 14 | ||||||
11 | An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques | "Browne, Glenn J.","Rogich, Michael B." | 2001 | Eliciting requirements from users and other stakeholders is of central importance to information systems development. Despite this importance, surprisingly little research has measured the effectiveness of various requirements elicitation techniques. The present research first discusses theory relevant to information requirements determination in general and elicitation in particular.We then develop a model of the requirements elicitation process. This model and its underlying theory were then used to construct a new requirements elicitation prompting technique. To provide a context for testing the relative effectiveness of the new technique, two other questioning methodologies were also operationalized as prompting techniques: (1) the interrogatories technique, which involves asking “who,” “what,” “when,” “where,” “how,” and “why” questions; and (2) a semantic questioning scheme, which involves asking questions based on a theoretical model of knowledge structures. To measure the usefulness of the prompting techniques in eliciting requirements, a set of generic requirements categories was adapted from previous research to capture requirements evoked by users. The effectiveness of the three methods in eliciting requirements for a software application was then tested in an experiment with users. Results showed that the new prompting technique elicited a greater quantity of requirements from users than did the other two techniques. Implications of the findings for research and systems analysis practice are discussed. | G. J. Browne and M. B. Rogich, An Empirical Investigation of User Requirements Elicitation: Comparing the Effectiveness of Prompting Techniques, J. Manage. Inf. Syst., vol. 17, no. 4, pp. 223–249, Mar. 2001, doi: 10.1080/07421222.2001.11045665. | https://doi.org/10.1080/07421222.2001.11045665 | Theory/Model | Compares techniques for extracting user requirements in analysis tasks. | Compares the methodologies for eliciting requirments from users. Compared Interview techniques: Syntactic Prompting (5w's - Who what where why and how) Semantic Prompting (Goals, Events, Agents, Actions, States) Tasks Characteritics (Scenario building, Conditionalizing, Elaborating with Instances, Hedging, Generating Conuterarguments, Geneerating Arguments, Feedback, Summarization) Elicited the requirments for a Online Grocery shopping expereince - something that never exsisted before but they have familiarity with in a different context. | 45 nonfaculty emplyees with expereince using databases | RQ2-Process | Identified that Prompting Techniques are better for extracting user requirments. | Determined the "effectiveness of a prompting technique in the elicitation of require" | Methodology | 15 | ||||||
12 | A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide? | "Dimara, Evanthia","Stasko, John" | 2021 | It has been widely suggested that a key goal of visualization systems is to assist decision making, but is this true? We conduct a critical investigation on whether the activity of decision making is indeed central to the visualization domain. By approaching decision making as a user task, we explore the degree to which decision tasks are evident in visualization research and user studies. Our analysis suggests that decision tasks are not commonly found in current visualization task taxonomies and that the visualization field has yet to leverage guidance from decision theory domains on how to study such tasks. We further found that the majority of visualizations addressing decision making were not evaluated based on their ability to assist decision tasks. Finally, to help expand the impact of visual analytics in organizational as well as casual decision making activities, we initiate a research agenda on how decision making assistance could be elevated throughout visualization research. | E. Dimara and J. Stasko, A Critical Reflection on Visualization Research: Where Do Decision Making Tasks Hide?, IEEE Trans. Visual. Comput. Graphics, pp. 1–1, 2021, doi: 10.1109/TVCG.2021.3114813. | https://ieeexplore.ieee.org/document/9552846/ | Review_ | Decision making tasks are missing from vis papers - why is that? | Argues that most visualization help with the understanding the data ( eg. the Intelligence stage of the simon model) but fail to incorporate decision making models. Research Agenda: Leverage other domain experties, expand methodologoes Clarify tasks and Calibrate Claims Equip visualization designers with decision making guidelines. Visualization research collectively advocates that decision making is, or should be, a core goal of visualization. We revisited visualization history, theory and practice to find that, in fact, visualization largely lacks explicit ties to decision making. V | Literature Review | RQ2-Process | Critical need for decision making studies in visualization research. | We have room to ask analysts to describe their desicion making process | Research Methodologies | 16 | ||||||
13 | Survey on Individual Differences in Visualization | "Liu, Zhengliang","Crouser, R. Jordan","Ottley, Alvitta" | 2020 | Developments in data visualization research have enabled visualization systems to achieve great general usability and application across a variety of domains. These advancements have improved not only people’s understanding of data, but also the general understanding of people themselves, and how they interact with visualization systems. In particular, researchers have gradually come to recognize the deficiency of having one-size-fits-all visualization interfaces, as well as the significance of individual differences in the use of data visualization systems. Unfortunately, the absence of comprehensive surveys of the existing literature impedes the development of this research. In this paper, we review the research perspectives, as well as the personality traits and cognitive abilities, visualizations, tasks, and measures investigated in the existing literature. We aim to provide a detailed summary of existing scholarship, produce evidence-based reviews, and spur future inquiry. | Z. Liu, R. J. Crouser, and A. Ottley, Survey on Individual Differences in Visualization, Computer Graphics Forum, vol. 39, no. 3, pp. 693–712, 2020, doi: 10.1111/cgf.14033. | https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.14033 | Review_ | A review of personal differences in visualization research | STAR Report: Looks at personality traits, Cognitive abilities, visualizations, tasks, and measures | Literature Review | RQ2-Process | May be worth looking at how literature capture individual differences and personalize visualizations. We may want to adopt similar methods | Research Methodologies | 17 | |||||||
14 | The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis | "Pirolli, Peter","Card, Stuart" | 2005 | There are a relatively few open literature reports that provide empirical descriptive studies of intelligence analysis and that link these into the context of expertise and work. This paper, based on first results from a cognitive task analysis and verbal protocols give a broad brush description of intelligence analysis as an example of sensemaking. It then suggests some possible leverage points where technology might be applied. | P. Pirolli and S. Card, The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis, presented at the International Conference on Intelligence Analysis, Jan. 2005. | https://www.researchgate.net/publication/215439203_The_sensemaking_process_and_leverage_points_for_analyst_technology_as_identified_through_cognitive_task_analysis | Theory/Model | A model of how people make sense of complex information | Two main loops for describing how people work with data. Foraging loop (gathering data) and sensemaking loop (structuring data) The process is highly dynamic, nonlinear, and cyclical in nature. | Interviews with a handful of real analysts | RQ2-Process | Describes a model of how analyusts think through information in their day to day | A key model about sense making we should be aware of. | Modeling Interactions | 18 | ||||||
15 | Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes | "Ragan, Eric D.","Endert, Alex","Sanyal, Jibonananda","Chen, Jian" | 2016 | While the primary goal of visual analytics research is to improve the quality of insights and findings, a substantial amount of research in provenance has focused on the history of changes and advances throughout the analysis process. The term, provenance, has been used in a variety of ways to describe different types of records and histories related to visualization. The existing body of provenance research has grown to a point where the consolidation of design knowledge requires cross-referencing a variety of projects and studies spanning multiple domain areas. We present an organizational framework of the different types of provenance information and purposes for why they are desired in the field of visual analytics. Our organization is intended to serve as a framework to help researchers specify types of provenance and coordinate design knowledge across projects. We also discuss the relationships between these factors and the methods used to capture provenance information. In addition, our organization can be used to guide the selection of evaluation methodology and the comparison of study outcomes in provenance research. | E. D. Ragan, A. Endert, J. Sanyal, and J. Chen, Characterizing Provenance in Visualization and Data Analysis: An Organizational Framework of Provenance Types and Purposes, IEEE Transactions on Visualization and Computer Graphics, vol. 22, no. 1, pp. 31–40, Jan. 2016, doi: 10.1109/TVCG.2015.2467551. | https://ieeexplore.ieee.org/abstract/document/7192714 | Theory/Model | A framework of the term: “Provenance" Help summarize and guide research in Visual Analytics | Existing perspectives: Provenance in Workflow Management Follow the data Graphical History Interaction History Sensemaking and insight FRAMEWORK: Types of Provenance Information - Data (changes/movement to data) - Visualization (different view states) - Interactions (history of user actions) - Insights (cognitive outcomes from Analysis) - Rationale (user intentions and reasoning over time) Purpose of Provenance - Recall (generic obvious option) - What’s been done, what are the findings, and what remains - Replication (application of recall to verify and validate other analyst work) - Action Recovery (restore previous states) - undo/redo - Collaborative Communication (help other people understand the state of analysis) - Presentation (communicate with non-analysts) - Meta-analysis (evaluate process efficiency and strategies. Recommendations: - capture More granularity and capture uncertainty (rationale and insights) we do not understand the effectiveness of uncertainty representations in the context of history and workflow visualizations - Limited work supporting rationale provenance, but plenty of support for the provenance of data, visualizations and interactions as well as for the purposes of recall, replication, and action recovery. - More specific evaluation of provenance information is needed. - Observations of users, user logs, controlled experiments, or heuristic evaluations and cognitive walkthroughs | Literature Review | RQ2-Process | RQ1-Info | This framework looks at provenance in the data science and visualization literature to better describe the ways provenance can serve to describe what work has been done. May be a good framework to describe the inputs and outputs in analyst processes. Applicable to RQ1-Info RQ2-Process | Modeling Interactions | 19 | ||||||
16 | Provectories: Embedding-based Analysis of Interaction Provenance Data | "Walchshofer, Conny","Hinterreiter, Andreas","Xu, Kai","Stitz, Holger","Streit, Marc" | 2021 | Understanding user behavior patterns and visual analysis strategies is a long-standing challenge. Existing approaches rely largely on time-consuming manual processes such as interviews and the analysis of observational data. While it is technically possible to capture a history of user interactions and application states, it remains difficult to extract and describe analysis strategies based on interaction provenance. In this paper, we propose a novel visual approach for meta-analysis of interaction provenance. We capture single and multiple user sessions as graphs of high-dimensional application states. Our meta-analysis is based on two different types of two-dimensional embeddings of these high-dimensional states: layouts based on (i) topology and (ii) attribute similarity. We applied these visualization approaches to synthetic and real user provenance data captured in two user studies. From our visualizations, we were able to extract patterns for data types and analytical reasoning strategies. | C. Walchshofer, A. Hinterreiter, K. Xu, H. Stitz, and M. Streit, Provectories: Embedding-based Analysis of Interaction Provenance Data, IEEE Transactions on Visualization and Computer Graphics. pp. 1–1, 2021. doi: 10.1109/TVCG.2021.3135697. | https://osf.io/mtfxn/ | Application/Design Study | Attempts to visualize analysis trajectories to aid in the analysis of analysis patterns | To support multi-session investigation, Provectories aims to: - M1 provides an overview of all the analysis sessions, such as which part of the dataset is more frequently investigated and where does most of the unsuccessful analysis ended up; - M2 support the comparison among analysis sessions, e.g., do successful analysis sessions share similar exploration pathways and if there is any common difference between successful and unsuccessful sessions; - M3 facilitates the discovery of other sense-making patterns, such as whether more efficient analysis sessions can be identified by certain visual patterns and is there any correlation between the investigation strategy and data attributes/subspace. Suggests that Provenance is a graph with states as notes and interactions as edges. - represents this visually. Represents the interaction histories as graphs and attributes(coverage) as ways of clustering interactions together in space) makes it easier to see similar states among users. Used Gap minder as an analysis tool and visualizes the interactions as vectors in space connected via a spline. | RQ2-Process | Methodology for evaluating the processies and making sense of interaction logs. | Modeling Interactions | 20 | ||||||||
17 | Do We Need “Teaming” to Team with a Machine? | Craig Haimson, Celeste Lyn Paul, Sarah Joseph, Randall Rohrer & Bohdan Nebesh | 2019 | What does it mean for humans and machines to work together effectively on complex analytic tasks? Is human teaming the right analogue for this kind of human-machine interaction? In this paper, we consider behaviors that would allow next-generation machine analytic assistants (MAAs) to provide context-sensitive, proactive support for human analytic work – e.g., awareness and understanding of a user’s current goals and activities, the ability to generate flexible responses to abstractly-formulated needs, and the capacity to learn from and adapt to changing circumstances. We suggest these behaviors will require processes of coordination and communication that are similar to but at least partially distinguishable from those observed in human teams. We also caution against over-reliance on human teaming constructs and instead advocate for research that clarifies the functions these processes serve in enabling joint activity and determines the best way to execute them in specific contexts. | Haimson, C., Paul, C.L., Joseph, S., Rohrer, R., Nebesh, B. (2019). Do We Need “Teaming” to Team with a Machine?. In: Schmorrow, D., Fidopiastis, C. (eds) Augmented Cognition. HCII 2019. Lecture Notes in Computer Science(), vol 11580. Springer, Cham. https://doi.org/10.1007/978-3-030-22419-6_13 | https://link.springer.com/chapter/10.1007/978-3-030-22419-6_13 | Theory/Model | Human Machine Teaming (HMT) does not need to mimic all behaviors of human teaming to be effective. There are specific behaviors that seem most relevant from an intelligence analysis perspective. | Machine Analytic Assistant (MAA) HMT will resemble human teaming but differ from it as well. The paper assesses five major factors that appear to affect the success of human teams: team leadership, mutual performance modeling, backup behavior, adaptability, and team orientation, along with the coordinating mechanisms of shared mental models, mutual trust, and closed-loop communication. | Literature Review | RQ2-Process | Discusses HMT in the context of intelligence analysis processes | "Is a more human-like machint teamate better than a machine teammate. Is it appropriate?" Explores the extent to which human teamwork is an appropriate model for the use case of intelligence analysis HMT | Methodology | 21 | ||||||
18 | Toward a visualization-supported workflow for cyber alert management using threat models and human-centered design | Franklin, Lyndsey,"Pirrung, Meg","Blaha, Leslie","Dowling, Michelle","Feng, Mi" | 2017 | Cyber network analysts follow complex processes in their investigations of potential threats to their network. Much research is dedicated to providing automated decision support in the effort to make their tasks more efficient, accurate, and timely. Support tools come in a variety of implementations from machine learning algorithms that monitor streams of data to visual analytic environments for exploring rich and noisy data sets. Cyber analysts, however, need tools which help them merge the data they already have and help them establish appropriate baselines against which to compare anomalies. Furthermore, existing threat models that cyber analysts regularly use to structure their investigation are not often leveraged in support tools. We report on our work with cyber analysts to understand the analytic process and how one such model, the MITRE ATT&CK Matrix [42], is used to structure their analytic thinking. We present our efforts to map specific data needed by analysts into this threat model to inform our visualization designs. We leverage this expert knowledge elicitation to identify a capability gaps that might be filled with visual analytic tools. We propose a prototype visual analytic-supported alert management workflow to aid cyber analysts working with threat models. | L. Franklin, M. Pirrung, L. Blaha, M. Dowling, and M. Feng, Toward a visualization-supported workflow for cyber alert management using threat models and human-centered design, in 2017 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2017, pp. 1–8. doi: 10.1109/VIZSEC.2017.8062200. | doi.org/10.1109/VIZSEC.2017.8062200 | Application/Design Study | Interviews Cyber analysts and Designs a VA alert managment workflow to handle threats | Phase1 - interviews about daily Workflows/tasks and general workflows Phase2 - Focus group - Asked analysts to match their workflow processes to the ATT&CK matrix (threat modeling / phases of attack) Identiefied gapes in Cyber Defender Support tools Designed a VA tool with inbox metaphor, Automated Machine Analytics, and interactive visualizations | 4 cyber analysts | RQ2-Process | they use interviews to elicit the workflows and regular duties | Elicit Design Requirements from Cyber analysts and design a tool that helps streamline workflows and facilitate alert triage | Does not evaluate the tool they make | Interview Study | 22 | |||||
19 | Critical Thinking and Intelligence Analysis | Moore, David T. | 2010 | Author David Moore makes a powerful argument that analysts who possess critical thinking skills are better able to cope with the complexities of a post-Cold War world than those who are not.Although technology can assist analysts by cataloguing and presenting data, information and evidence in new ways, it cannot do the analysis for them. To be most effective, analysts need an overarching, reflective framework to add structured reasoning to sound, intuitive thinking. Critical thinking provides such a framework and goes further, positively influencing the entire intelligence analysis process. This paper defines critical thinking in the context of intelligence analysis, explains how it influences the entire intelligence process, explores how it toughens the art of intelligence analysis, suggests how it may be taught, and deduces how analysts can be persuaded to adopt this habit. Related products: Crafting an Intelligence Community: Papers of the First Four DCIs (Book and DVD) is available here: https: //bookstore.gpo.gov/products/sku/041-015-00298-8Sensemaking: A Structure for an Intelligence Revolution (2012) is availalbe here: https: //bookstore.gpo.gov/products/sku/008-000-01062-5A Life in Intelligence: The Richard Helms Collection (Book and DVD) can be found here: https: //bookstore.gpo.gov/products/sku/041-015-00292-9 Who Watches the Watchmen?: The Conflict between National Security and Freedom of the Pressis available here: https: //bookstore.gpo.gov/products/sku/008-020-01606-3The FBI Story 2015 is available here: https: //bookstore.gpo.gov/products/sku/027-001-00102-1Intelligence & Espionage resources collection can be found here: https: //bookstore.gpo.gov/catalog/security-defense-law-enforcement/intelligence-espionage " | D. T. Moore, Critical Thinking and Intelligence Analysis. Government Printing Office, 2010. | https://apps.dtic.mil/sti/pdfs/ADA481702.pdf | Book | Discusses consequences of lack of critical thinking in intelligence analysis and proposes | 159 page "paper" published by National Defense Intelligence College | N/A | RQ2-Process | Describes how critical thinking does and should factor into intelligence analysis process | Establishes role and importance of critical thinking in intelligence analysis | Methodology | 23 | ||||||
20 | Information sharing and collaboration in the United States Intelligence community: an ethnographic study of the National Counterterrorism Center | Nolan, Bridget Rose | 2013 | The National Counterterrorism Center (NCTC) was established to serve as the primary organization in the U.S. Government for the integration, sharing, and analysis of all terrorism and counterterrorism intelligence. To date, no study has sought to illustrate whether and how NCTC overcomes the barriers to information sharing among agencies and the people that comprise them. The purpose of this dissertation is to explore the micro-level ways in which intelligence work is conducted in a post-9/11 world and to examine the circumstances that both facilitate and discourage collaboration. By presenting detailed ethnographic evidence and the in-depth interview perspectives of the people who actually do this work daily, this study provides a sociological analysis and discussion of best practices to help identify ways in which NCTC can move closer to fulfilling its mission. | Nolan, B. R. (2013). Information sharing and collaboration in the United States Intelligence community: an ethnographic study of the National Counterterrorism Center (Doctoral dissertation, University of Pennsylvania). | https://indianstrategicknowledgeonline.com/web/Nolan_Dissertation.pdf | Dissertation/Thesis | Deep qualitative analysis of a counter-terrorism analyst's work/life. RQs on p. 6-7 are very focused on collaboration | Key chapters: Chapter 2 - Daily life of a counterterrorism analyst from a micro-level perspective. Chapter 3 - Effects of status inequality among the agencies on inter-agency collaboration. Chapter 4 - Creating written products for policymakers. Research site was National Counterterrorism Center (NCTC). | CIA/NCTC analysts and other employees | RQ3-Job | RQ1-Info | This dissertation covers topics related to all 4 listed RQs in a fair amount of depth. It gives a very in depth view into life as an analyst and goes into detail about all of the things that make the job complicated: collaboration, perception of one's own agency and other agencies, the reporting process, information gathering and the effects of incompatible systems and information stovepiping | Comprehensive view into the challenges and realities of working as an intelligence analyst, suggestions for how to resolve identified problems | The two primary limitations here are: 1. This work focuses on CIA/NCTC which likely has cultural differences from NSA and 2. There is limited focus on technology | Observational Study | 24 | ||||
21 | Investigating the Role of Locus of Control in Moderating Complex Analytic Workflows | "Crouser, R. Jordan","Ottley, Alvitta","Swanson, Kendra","Montoly, Ananda" | 2020 | Throughout the last decade, researchers have shown that the effectiveness of a visualization tool depends on the experience, personality, and cognitive abilities of the user. This work has also demonstrated that these individual traits can have significant implications for tools that support reasoning and decision-making with data. However, most studies in this area to date have involved only short-duration tasks performed by lay users. This short paper presents a preliminary analysis of a series of exercises with 22 trained intelligence analysts that seeks to deepen our understanding of how individual differences modulate expert behavior in complex analysis tasks. | R. J. Crouser, A. Ottley, K. Swanson, and A. Montoly, Investigating the Role of Locus of Control in Moderating Complex Analytic Workflows, EuroVis 2020 - Short Papers, p. 5 pages, 2020, doi: 10.2312/EVS.20201050. | https://diglib.eg.org/handle/10.2312/evs20201050 | Application/Design Study | While personality factors could be attributed to personalized interfaces, most visualization papers focus on a single analysis session and do not assess personality factors longitudinally. | we conducted a series of multi-day exercises with trained intelligence analysts to investigate their behavior during complex analysis tasks. Asked analysts to use an interface and track their notes in google docs. Looking at their locus of control and the interactions was interesting: 1. Interaction Volume - We observed that participants with a more internal LOC tended to perform more distinct actions with both the interface and in editing their notes than those whose LOC was more external 2. Data Coverage - Those with an internal LOC visit more of the data 3. Anchoring - "clicking down" behavior observed - Earlier for internal LOC and later for External LOC. | 15 Real analysts through LAS | RQ3-Job | Understanding the people and their work related to individual differneces in users. We could do psychological batteries to identify personal differencces in ananlysts and look for correllations in how analysts perceive their role and the work they produce. | Preliminary study into relationship with Locus of Control and analysis tasks. | Personality factors play a role in intelligence analysis and should be involved with the investigation. | Research Methodologies | 25 | |||||
22 | Analytic Provenance in Practice: The Role of Provenance in Real-World Visualization and Data Analysis Environments | "Madanagopal, Karthic","Ragan, Eric D.","Benjamin, Perakath" | 2019 | Practical data analysis scenarios involve more than just the interpretation of data through visual and algorithmic analysis. Many real-world analysis environments involve multiple types of experts and analysts working together to solve problems and make decisions, adding organizational and social requirements to the mix. We aim to provide new knowledge about the role of provenance for practical problems in a variety of analysis scenarios central to national security. We present the findings from interviews with data analysts from domains, such as intelligence analysis, cyber-security, and geospatial intelligence. In addition to covering multiple analysis domains, our study also considers practical workplace implications related to organizational roles and the level of analyst experience. The results demonstrate how different needs for provenance depend on different roles in the analysis effort (e.g., data analyst, task managers, data analyst trainers, and quality control analysts). By considering the core challenges reported along with an analysis of existing provenance-support techniques through existing research and systems, we contribute new insights about needs and opportunities for improvements to provenance-support methods. | K. Madanagopal, E. D. Ragan, and P. Benjamin, Analytic Provenance in Practice: The Role of Provenance in Real-World Visualization and Data Analysis Environments, IEEE Comput. Grap. Appl., vol. 39, no. 6, pp. 30–45, Nov. 2019, doi: 10.1109/MCG.2019.2933419. | https://ieeexplore.ieee.org/document/8788592/ | Theory/Model | Looking at how Provenance is used in the real world by interviewing experts. Recognizing that different roles have different needs | Surveyed analysts to better understand their problems and the types of provenance they interacted with on a daily basis. Analyst Ranked the priority of their issues to develop a set of tables as seen in the screenshots attached. Little research has been done in understanding different job duties performed by an analyst and their challenges in performing their day-to-day operations. We pri- marily focused on studying the challenges in per- forming various analysis tasks and whether provenance based solutions are helpful in per- forming their job duties. | Primarily intelligence analysis, geo-spatial-imagery analysis, and cyber-security analysis. - 4 general industry domain people too | RQ3-Job | Survey study to understand what challenges analysts in general face. They identified different roles, Problems, and ways provenance may support their work | research into the job duties perforned and challenged encounterd by an analyst | "Advancements are needed to support common work- flow needs" | These are areas of interest (that can be supported by provenance) as defined by analysts | Research Methodologies | 26 | ||||
23 | Voice of Experience: Principles of Intelligence Analysis | Robert Levine, PhD | 2021 | https://www.cia.gov/static/7514ffff9baca6c514433ee0f55629db/Article-Principles-of-Intelligence-Analysis-Studies65-4-Dec2021.pdf | The paper presents key principles for delivering value-added insights that are accurate, relevant, timely, and persuasive. | 1 Intelligence Analyst | RQ3-Job | A career analyst's perspective on the most important priciples of intelligence analysis | Compiles principles of intelligence analysis from an experienced analyst. | Only one analyst's perspective. | 27 | ||||||||||
24 | Developing a Taxonomy of Intelligence Analysis Variables | Johnston, Rob | 2003 | Aristotle may be the father of scientific classification, but it was Carolus Linnaeus who introduced the first formal taxonomy kingdom, class, order, genera, and species-in his Systema Naturae in 1735. By codifying the naming conventions in biology, Linnaeuss work provided a reference point for future discoveries. | R. Johnston, Developing a Taxonomy of Intelligence Analysis Variables, CENTRAL INTELLIGENCE AGENCY WASHINGTON DC CENTER FOR THE STUDY OF INTELLIGENCE, Jan. 2003. Accessed: Jun. 29, 2022. [Online]. Available at: https://apps.dtic.mil/sti/citations/ADA525551 | https://apps.dtic.mil/sti/citations/ADA525551 | Theory/Model | Develops a taxonomy for intelligence analysis | Taxonomy consists of 4 high-level categories: systemic, systematic, idiosyncratic, communicative. Each of these roughly pertain to one of our research questions regarding information, process, work environment, and collaboration. | No sample population but paper is written from intelligence analyst perspective for IC | RQ3-Job | This helps to model the ecosystem, mentally and physically, of an intelligence analyst | Presented taxonomy may help us in using correct language when we talk to analysts and identifying interview/research areas | This isn't empirical work | Methodology | 28 | |||||
25 | Mixed method approach to identify analytic questions to be visualized for military cyber incident handlers | Buchanan, Laurin,"D’Amico, Anita","Kirkpatrick, Drew" | 2016 | Our multi-disciplinary team developed and applied a 6-step mixed method approach to efficiently identify the cognitive work of early stage military cyber incident handlers, extract a subset of that work that could benefit from visualization, and specify the information needs as Analytic Questions (AQs) posed by operators that the visualizations would have to support. The methodology included a survey of subject matter experts to validate that the major findings of prior research on the cognitive work of cyber defenders, conducted over a decade ago, are still valid today. It also utilized a Goal Directed Task Analysis (GDTA) structure to represent the major task, goals, decisions, AQs and data source requirements of early stage cyber incident handlers. This yielded 40 AQs which are reported in this paper. Knowledge Elicitation (KE) interviews of domain practitioners were used to select the AQs with greatest potential for incorporation into a follow-on project to measure the effects of visualization on early stage incident handler performance. The AQs represent measurable units of cognitive work which must be performed using available data in a severely time-constrained work environment. Thus, they can serve as indicators of operator performance to be used in experiments on the effectiveness of visualization for event detection and preliminary analysis. They can also provide requirements for visualization designers and security products. | L. Buchanan, A. D’Amico, and D. Kirkpatrick, Mixed method approach to identify analytic questions to be visualized for military cyber incident handlers, in 2016 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2016, pp. 1–8. doi: 10.1109/VIZSEC.2016.7739578. | doi.org/10.1109/VIZSEC.2016.7739578 | Technique | Applied a mixed method approch to find common questions cyber analysts need to answer. | Surveyed the literature, found the 10-year old issues are still valid conducted a Goal Directed Task Analysis to find 40 Analyst Quetions Conducted interviews with practicioners to select key AQ's to address with Visualization Suggests using AQ's as indicators of performance. Discusses the Cyber Incident Handling Life Cycle [18]. | Literature & Cyber analysts | RQ3-Job | similar to the work we are doing, they try to understand commonalities among cyber analyst workflows. | Ideentifies "early stage" challenges that could be improved for cyber analysts with visualization. | They only talk to cyber analysts. | Methodology | 29 | |||||
26 | Supporting Handoff in Asynchronous Collaborative Sensemaking Using Knowledge-Transfer Graphs | Zhao, Jian,"Glueck, Michael","Isenberg, Petra","Chevalier, Fanny","Khan, Azam" | 2018 | During asynchronous collaborative analysis, handoff of partial findings is challenging because externalizations produced by analysts may not adequately communicate their investigative process. To address this challenge, we developed techniques to automatically capture and help encode tacit aspects of the investigative process based on an analyst’s interactions, and streamline explicit authoring of handoff annotations. We designed our techniques to mediate awareness of analysis coverage, support explicit communication of progress and uncertainty with annotation, and implicit communication through playback of investigation histories. To evaluate our techniques, we developed an interactive visual analysis system, KTGraph, that supports an asynchronous investigative document analysis task. We conducted a two-phase user study to characterize a set of handoff strategies and to compare investigative performance with and without our techniques. The results suggest that our techniques promote the use of more effective handoff strategies, help increase an awareness of prior investigative process and insights, as well as improve final investigative outcomes. | J. Zhao, M. Glueck, P. Isenberg, F. Chevalier, and A. Khan, Supporting Handoff in Asynchronous Collaborative Sensemaking Using Knowledge-Transfer Graphs, IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 1, pp. 340–350, Jan. 2018, doi: 10.1109/TVCG.2017.2745279. | doi.org/10.1109/TVCG.2017.2745279 | System | Development and evaluation of a visual analytics system to coordinate asychronous handoff of investigative document analysis | Developed a system KTGraph (knowledge-transfer graph) to assist handoff of investigative document analysis with goals: 1. interactivity and flexible representation of thought processes/investigative styles, 2. built-in analytic provenance, 3. support common understanding, 4. support provenance of the investigative process | Grad students and researchers in compsci or engineering | RQ4-Collab | The system presented here is intended to aid collaboration in the analysis process by making analysis artifacts + activities visible and accessible to multiple users | Identification of important factors for collaboration on analysis, development of system for asynchronous handoff of analysis, evaluation of system | Participants were not analysts (grad students), system design seems to assume that analysts would work together on an investigation and that there is not an issue of different clearance/access to different information | Tools/Systems | 30 | |||||
27 | Collaborative synthesis of visual analytic results | Robinson, Anthony C. | 2008 | Visual analytic tools allow analysts to generate large collections of useful analytical results. We anticipate that analysts in most real world situations will draw from these collections when working together to solve complicated problems. This indicates a need to understand how users synthesize multiple collections of results. This paper reports the results of collaborative synthesis experiments conducted with expert geographers and disease biologists. Ten participants were worked in pairs to complete a simulated real-world synthesis task using artifacts printed on cards on a large, paper-covered workspace. Experiment results indicate that groups use a number of different approaches to collaborative synthesis, and that they employ a variety of organizational metaphors to structure their information. It is further evident that establishing common ground and role assignment are critical aspects of collaborative synthesis. We conclude with a set of general design guidelines for collaborative synthesis support tools. | A. C. Robinson, Collaborative synthesis of visual analytic results, in 2008 IEEE Symposium on Visual Analytics Science and Technology, Oct. 2008, pp. 67–74. doi: 10.1109/VAST.2008.4677358. | doi.org/10.1109/VAST.2008.4677358 | Application/Design Study | Characterize and design for collaborative result synthesis. | Participants collaboratively worked on an information synthesis activity based off of a military intelligence training activity. | Expert geographers and disease biologists | RQ4-Collab | Identifies important elements of collaborative information synthesis to highlight potential features for a collaborative tool for analysts | Identified important aspects of collaborative information synthesis in the analytic process + presented resulting implications for design of visual analytic systems | Participants were not analysts (postdocs and PhD candidates) | Observational Study | 31 | |||||
28 | Human-Human Communication in Cyber Threat Situations: A Systematic Review | Torvald F. Ask, Ricardo G. Lugo, Benjamin J. Knox and Stefan Sutterlin | 2021 | In cyber threat situations, decision-making processes within organizations and between the affected organization and external entities are high-stake. They require human communication entailing technical complexity, time pressure, interdisciplinary factors, and often an insufficient information basis. Communication in cyber threat situations can thus be challenging and has a variety of implications for decision-making. The cyber-physical system is a rapidly changing socio-technical system that is understudied in terms of how cyber events are communicated and acted upon to secure and maintain cyber resilience. The present study is the first to review human-to-human communication in cyber threat situations. Our aims are to outline how human-human communication performance in cybersecurity settings have been studied, to uncover areas where there is potential for developing common standards for information exchange in collaborative settings, and to provide guidance for future research efforts. The review was carried out according to the PRISMA guidelines and articles were searched for on scientific databases. Articles focusing on human-human communication in cyber threat situations published in peer reviewed journals or as conference papers were included. A total of 17 studies were included in the final review. Most of the studies were correlational and exploratory in nature. Very few studies characterize communication in useful goal-related terms. There is a need for more collaboration between cyber defense exercise-organizers and cognitive scientists. Future studies should assess how team mental model-development affects team communication and performance in cyber defense exercises. | Ask, T. F., Lugo, R. G., Knox, B. J., & Sütterlin, S. (2021, July). Human-Human Communication in Cyber Threat Situations: A Systematic Review. In International Conference on Human-Computer Interaction (pp. 21-43). Springer, Cham. | https://link.springer.com/chapter/10.1007/978-3-030-90328-2_2 | Review_ | In Security Operation Centers (SOCs), teechnical staff usually does the asset monitoring, detection, analysis, forensics, etc while there are different decision-makers. There exists a potential knowledge gap between these two entities. This paper reviews existing literature about human-human communication of cyber threat information. | The goals of the paper are to outline how human-human communication performance in cybersec has been studied, to uncover areas where there is potential for developing COMMON STANDARDS for info exchange in collaborative settings, and to provide guidance for future research efforts. A review of 17 studies in total. | RQ4-Collab | 32 | ||||||||||
29 | The Group Matters: A review of processes and outcomes in intelligence analysis | Susan G. Straus, Andrew M. Parker, James B. Bruce | 2011 | The work of intelligence analysts is fundamentally cognitive in nature. Intelligence analysis consists largely of identifying problems, generating and evaluating hypotheses, identifying and assessing open source and classified information, recognizing patterns in large sets of data, aggregating information, and providing results in the form of judgments, forecasts, and insights to policymakers. These activities are often conducted by individuals; however, intelligence agencies and experts have called increasingly for the use of teams in intelligence analysis. This article reviews the research literature on group-level phenomena (that is, process losses) that are most relevant to the work of intelligence analysts, including productivity losses in brainstorming, the common knowledge effect, group polarization, confirmation bias, overconfidence, and pressures toward uniformity. We describe how features of intelligence analysis teams' tasks, context, and structure affect these processes, present methods to minimize these process losses and increase process gains, and discuss directions for future research. Although our focus is on intelligence analysis teams, these processes and interventions are relevant to a range of analytical teams that share common characteristics. | Straus, S. G., Parker, A. M., & Bruce, J. B. (2011). The group matters: A review of processes and outcomes in intelligence analysis. Group Dynamics: Theory, Research, and Practice, 15(2), 128. | https://psycnet.apa.org/record/2011-07687-001?doi=1 | Review_ | RQ4-Collab | 33 | ||||||||||||
30 | Interdisciplinary , cross-sector collaboration in the US Intelligence Community: lesson learned from past and present efforts | Kathleen M. Vogel and Beveryly B. Tyler | 2019 | How does one design and sustain interdisciplinary, cross-sector collaboration to improve intelligence results for twenty-first century security threats? This paper will analyse five past and present initiatives designed to create interdisciplinary, cross-sectoral collaboration within different agencies of the US Intelligence Community (IC). We will discuss key features of each effort, their successes and challenges, identify common themes and, propose which collaborative model might be most advantageous for a particular type of project based on project constraints. In so doing, we provide direction for IC leaders seeking to improve academia–industry–intelligence partnerships for future planning on intelligence-funded collaborations. | Vogel, K. M., & Tyler, B. B. (2019). Interdisciplinary, cross-sector collaboration in the US Intelligence Community: lessons learned from past and present efforts. Intelligence and National Security, 34(6), 851-880. | https://www.tandfonline.com/doi/abs/10.1080/02684527.2019.1620545 | Theory/Model | RQ4-Collab | 34 | ||||||||||||
31 | A Framework for Thinking about Collaboration within the Intelligence Community | Joan McIntyre, Douglas Palmer, Justin Franks | 2009 | The Director for National Intelligence (DNI) envisions a globally networked and integrated intelligence enterprise created by integrating foreign, military, and domestic capabilities through policy, personnel and technology actions to provide decision advantage to policy makers, warfighters, homeland security officials and law enforcement personnel.i The DNI Vision 2015 states that to meet the demands for greater forethought and strategic agility the Intelligence Community must “evolve into a true Intelligence enterprise established on a collaborative foundation of shared services, mission-centric operations, and integrated mission management, all enabled by the smooth flow of people, ideas, and activities across the boundaries of the Intelligence Community members.”ii Underlying this vision is the goal to create a culture of collaboration and an integrated Intelligence enterprise. | McIntyre, J., Palmer, D., & Franks, J. (2009). A framework for thinking about collaboration within the intelligence community. Pherson & Associates, LLC, 10. | http://www.pherson.org/wp-content/uploads/2013/11/09.-A-Framework-for-Thinking-about-Collaboration-within-the-Intelligence-Community_FINAL.pdf | Theory/Model | Gives a general overview of existing collaborative practices and suggests a few ways to facilitate it better (with no real grounding to the work). | Key terms: Collaboration is not just information sharing. For effective collaboration, the same body of information is a necessary precondition. A need for horizontal integration (across departments and organizations) rather than traditional vertical (hierarchy). Netcentricity, essentially fosucing on underlying networks that connect individuals and facilitate communications and trust. Communities of interest - groups of people linked by technology and informally bound together by a common mission and passion for joint enterprise. A key point to stress is that collaborative technologies and tools can enable collaboration between individuals and groups separated by time and distance, but by themselves are unable to create the conditions conducive to effective collaboration. Types of collaboration: A team that has formally been tasked to work together vs "tacit" collab that occurs when individuals make their thoughts and results of their work available for others to respond or build on without formally coalescing into a team. | RQ4-Collab | Slightly relevant. No real implications. | Proposing technologies to facilitate research | No research. | Observational Study | 35 | ||||||
32 | Constraints on intelligence collaboration: The domestic dimension | James J. Wirtz | 2008 | Wirtz, J. J. (1993). Constraints on intelligence collaboration: The domestic dimension. International Journal of Intelligence and Counter Intelligence, 6(1), 85-99. | https://www.tandfonline.com/doi/pdf/10.1080/08850609308435203?needAccess=true | RQ4-Collab | 36 | ||||||||||||||
33 | InsideInsights: Integrating Data-Driven Reporting in Collaborative Visual Analytics | "Mathisen, A.","Horak, Tom","Klokmose, C. N.","Grønbæk, Kaj","Elmqvist, Niklas" | 2019 | Analyzing complex data is a non-linear process that alternates between identifying discrete facts and developing overall assessments and conclusions. In addition, data analysis rarely occurs in solitude; multiple collaborators can be engaged in the same analysis, or intermediate results can be reported to stakeholders. However, current data-driven communication tools are detached from the analysis process and promote linear stories that forego the hierarchical and branching nature of data analysis, which leads to either too much or too little detail in the final report. We propose a conceptual design for integrated data-driven reporting that allows for iterative structuring of insights into hierarchies linked to analytic provenance and chosen analysis views. The hierarchies become dynamic and interactive reports where collaborators can review and modify the analysis at a desired level of detail. Our web-based InsideInsights system provides interaction techniques to annotate states of analytic components, structure annotations, and link them to appropriate presentation views. We demonstrate the generality and usefulness of our system with two use cases and a qualitative expert review. | https://onlinelibrary.wiley.com/doi/10.1111/cgf.13717 | Application/Design Study | supporting the hierarchical nature of analysis by capturing full-fidelity insights and making them into an interactive presentation. | Other tools (Observable and Jupyter) do not allow for exploration and presentation - Also Slideshows and notebooks are linear presentations. - Level of detail is the point in those other tools. This is a hierarchical Insight management system that: - Coalesce findings into higher-level abstractions - Subdivide composite assumptions into validatable items results in a multi-level report that describes the pyramidal thinking and process of an analyst. Data-Driven Reporting Bottom-up - Foraging > Schemas > findings > presentations ^ iterative and branching always, thus presentation should be part of the analysis process. As analyzing the user flags system states to be part of the Presentation layer. try it out- http://github.com/90am/insideinsights video - https://vimeo.com/337474793 hierarchical structures can better support overview and comprehension, especially for readers with low prior knowledge of the content [ATM09,SVLV16]. In contrast, graph structures are better for certain information seeking tasks for readers with higher prior knowledge, but this structure imposes a more demanding process on the reader [ATM09,SVLV16]. | expert review - Associate Professsor in CS with 15 years Exp. industry visual analysis specialist with a PhD | RQ4-Collab | Focusing of the aspects of collaboration and helping others stay informed about insights as they develop. Directly discusses techniques for what a insight ecosystem would look like. | They develop an analysis ecosystem that tracks insights and aids in the presentation of these results. | No formal user studies. Just had an expert review PHASE 1 - asked experts to interpreate the generated report, make a change and identify the key events in the data set. PHASE 2 - Review the data analysis pipeline (filting.clustering etc.) and write their on narration. | Visualization/presentation techniques | 37 | ||||||
34 | Unlocking user-centered design methods for building cyber security visualizations | Mckenna, Sean,"Staheli, Diane","Meyer, Miriah" | 2015 | User-centered design can aid visualization designers to build better, more practical tools that meet the needs of cyber security users. The cyber security visualization research community can adopt a variety of design methods to more efficiently and effectively build tools. We demonstrate how previous cyber visualization research has omitted a discussion of effectiveness and process in the explanation of design methods. In this paper, we discuss three design methods and illustrate how each method informed two real-world cyber security visualization projects which resulted in successful deployments to users. | S. Mckenna, D. Staheli, and M. Meyer, Unlocking user-centered design methods for building cyber security visualizations, in 2015 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2015, pp. 1–8. doi: 10.1109/VIZSEC.2015.7312771. | doi.org/10.1109/VIZSEC.2015.7312771 | Technique | Recommends the need for Participatory Design in Cyber security visualization designs | Introduces Qualitative coding, Personas, and Data Sketches methodology as useful for desining tools for cyber security analysts. Demonstrate and report on each method's usefulness. | conducted interviews for 6 weeks with various stakeholders (network analysts, managers, researchers embedded in cyber operations, and various other cyber security and business-focused users) | RQ4-Collab | The develop a set of personas and draw a diagram that decribes who talks to whom in the workflow | Participatory design is helpful and should be used more frequently when designing visualizations for cyber securoty experts | Did not build an interactive interface. All individual interviews | Methodology | 38 | |||||
35 | Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics | Arias-Hernandez, Richard,"Kaastra, Linda T.","Green, Tera M.","Fisher, Brian" | 2011 | Studying how humans interact with abstract, visual representations of massive amounts of data provides knowledge about how cognition works in visual analytics. This knowledge provides guidelines for cognitive-aware design and evaluation of visual analytic tools. Different methods have been used to capture and conceptualize these processes including protocol analysis, experiments, cognitive task analysis, and field studies. In this article, we introduce Pair Analytics: a method for capturing reasoning processes in visual analytics. We claim that Pair Analytics offers two advantages with respect to other methods: (1) a more natural way of making explicit and capturing reasoning processes and (2) an approach to capture social and cognitive processes used to conduct collaborative analysis in real-life settings. We support and illustrate these claims with a pilot study of three phenomena in collaborative visual analytics: coordination of attention, cognitive workload, and navigation of analysis. | R. Arias-Hernandez, L. T. Kaastra, T. M. Green, and B. Fisher, Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics, in 2011 44th Hawaii International Conference on System Sciences, Jan. 2011, pp. 1–10. doi: 10.1109/HICSS.2011.339. | doi.org/10.1109/HICSS.2011.339 | Technique | Proposes Pair Analytics as a method of capturing reasoning processes and social and cognitive processes associated with collaborative analysis | 6 SMEs in aircraft engineering and visual anlytics | RQ4-Collab | Proposes a new method for capturing collaborative analytic process | Pair Analytics is a valuable method of capturing individual and social cognitive analytic processes and has two primary benefits. 1. It is a more natural way of capturing reasoning 2. It can capture social cognitive processes as well as individual | Paper is not really targeted towards IA, work is not that relevant to our interests | Methodology | 39 | ||||||
36 | Do visualizations improve synchronous remote collaboration? | Balakrishnan, Aruna D.,"Fussell, Susan R.","Kiesler, Sara" | 2008 | Information visualizations can improve collaborative problem solving, but this improvement may depend on whether visualizations promote communication. In an experiment on the effect of network visualizations, remote pairs worked synchronously to identify a serial killer. They discussed disparate evidence distributed across the pair using IM. Four conditions, respectively, offered (a) spreadsheet only (controls), (b) individual unshared visualizations, (c) view-only shared visualizations, and (d) a full-access shared visualization of all evidence. We examined collaborative performance, use of the visualization tool, and communication as a function of condition. All visualization conditions improved remote collaborators’ performance over the control condition. Full access to a shared visualization best facilitated remote collaboration by encouraging tool use and fostering discussion between the partners. Shared visualization without full access impaired performance somewhat and made communication even more vital to identifying the serial killer. This study provides direct evidence of visualization tool features and partner behavior that promote collaboration. | A. D. Balakrishnan, S. R. Fussell, and S. Kiesler, Do visualizations improve synchronous remote collaboration?, in Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI ’08, Florence, Italy, 2008, p. 1227. doi: 10.1145/1357054.1357246. | http://portal.acm.org/citation.cfm?doid=1357054.1357246 | Evaluation | Evaluates the impacts of four different visual analytic conditions on collaborative analysis | Four visual analytic conditions were compared: (a) spreadsheet only (controls), (b) individual unshared visualizations, (c) view-only shared visualizations, and (d) a full-access shared visualization of all evidence | 94 participants - mostly undergrad/grad students | RQ4-Collab | Evaluates how different interfaces/visual analytic techniques improve collaborative analysis | All visual analytic conditions performed better than the control but the best one was the full access to a shared visualization | Analytic focus is on law enforcement/crime investigation. Population is random, not intended to be proxy for IA | Observational Study | 40 | |||||
37 | Survey on the Analysis of User Interactions and Visualization Provenance | "Xu, Kai","Ottley, Alvitta","Walchshofer, Conny","Streit, Marc","Chang, Remco","Wenskovitch, John" | 2020 | There is fast-growing literature on provenance-related research, covering aspects such as its theoretical framework, use cases, and techniques for capturing, visualizing, and analyzing provenance data. As a result, there is an increasing need to identify and taxonomize the existing scholarship. Such an organization of the research landscape will provide a complete picture of the current state of inquiry and identify knowledge gaps or possible avenues for further investigation. In this STAR, we aim to produce a comprehensive survey of work in the data visualization and visual analytics field that focus on the analysis of user interaction and provenance data. We structure our survey around three primary questions: (1) WHY analyze provenance data, (2) WHAT provenance data to encode and how to encode it, and (3) HOW to analyze provenance data. A concluding discussion provides evidence-based guidelines and highlights concrete opportunities for future development in this emerging area. The survey and papers discussed can be explored online interactively at https://provenance-survey.caleydo.org. | K. Xu, A. Ottley, C. Walchshofer, M. Streit, R. Chang, and J. Wenskovitch, Survey on the Analysis of User Interactions and Visualization Provenance, Computer Graphics Forum, vol. 39, no. 3, pp. 757–783, Jun. 2020, doi: 10.1111/cgf.14035. | https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14035 | Review_ | STAR report on Provenance - Robust and important starting place. | website: https://provenance-survey.caleydo.org WHY - look at the provenance? Adaptive Systems, Evaluation of Systems and Algorithms, Model Steering and Active Learning, Replication, Report Generation, Understanding the User WHAT - is used to explore provenance? Grammar, Graph, Model, Sequence HOW - is the provenance visualized? Classification Models, Pattern Analysis, Probabilistic Models/Prediction, Program Synthesis, Interactive Visual Analysis | Literature Review | Comprehensive Literature Review about how provenance is used | Modeling Interactions | 42 | ||||||||
38 | Problem characterization and abstraction for visual analytics in behavior-based malware pattern analysis | Wagner, Markus,"Aigner, Wolfgang","Rind, Alexander","Dornhackl, Hermann","Kadletz, Konstantin","Luh, Robert","Tavolato, Paul" | 2014 | Behavior-based analysis of emerging malware families involves finding suspicious patterns in large collections of execution traces. This activity cannot be automated for previously unknown malware families and thus malware analysts would benefit greatly from integrating visual analytics methods in their process. However existing approaches are limited to fairly static representations of data and there is no systematic characterization and abstraction of this problem domain. Therefore we performed a systematic literature study, conducted a focus group as well as semi-structured interviews with 10 malware analysts to elicit a problem abstraction along the lines of data, users, and tasks. The requirements emerging from this work can serve as basis for future design proposals to visual analytics-supported malware pattern analysis. | M. Wagner et al., Problem characterization and abstraction for visual analytics in behavior-based malware pattern analysis, in Proceedings of the Eleventh Workshop on Visualization for Cyber Security, New York, NY, USA, Nov. 2014, pp. 9–16. doi: 10.1145/2671491.2671498. | https://doi.org/10.1145/2671491.2671498 | Technique | systematic literature review, focus group as well as semi-structured interviews with 10 malware detection analysts to find patterns in behaviors | Run focus groups to find patterns in tools Ran interviews to more fully understand individual problems. | The focus group consisted of 7 people: 4 IT-security experts and 3 visual analytics experts. Interviews with 10 (all the same from Focus group+ 3 new Industry experts) | RQ1-Info | RQ2-Process | Looks at the tools used and the kinds of work it security people do. | How do cyber analysts think about visualization tools for malware detection tools Lays out a model of how Cyber analysts work - work steps used for differnet tasks Define requirments for future VA tools to suport cyber traces | Only IT-security people. Only looked at exsisting tools at the time Groups identified tools of interest, Individuals reported on expereince with tools in interview | Methodology | 44 | ||||
39 | Analytic Culture in the US Intelligence Community: An Ethnographic Study | Johnston, Rob | 2005 | Identifies and describes conditions and variables that negatively affect intelligence analysis. Investigates analytic culture, methodology, error, and failure within the Intelligence Community. Uses an applied anthropological methodology that includes interviews, direct and participant observation, and focus groups. Contains a bibliography. | R. Johnston, Analytic Culture in the US Intelligence Community: An Ethnographic Study. Center for the Study of Intelligence, Central Intelligence Agency, 2005. | Compared various interfaces for user | Book | This is pretty old - published in 2005 and observations began shortly after 9/11 | Observational Study | 45 | |||||||||||
40 | Jigsaw: Supporting Investigative Analysis through Interactive Visualization | Stasko, John,"Görg, Carsten","Liu, Zhicheng" | 2008 | Investigative analysts who work with collections of text documents connect embedded threads of evidence in order to formulate hypotheses about plans and activities of potential interest. As the number of documents and the corresponding number of concepts and entities within the documents grow larger, sense-making processes become more and more difficult for the analysts. We have developed a visual analytic system called Jigsaw that represents documents and their entities visually in order to help analysts examine them more efficiently and develop theories about potential actions more quickly. Jigsaw provides multiple coordinated views of document entities with a special emphasis on visually illustrating connections between entities across the different documents. | J. Stasko, C. Görg, and Z. Liu, Jigsaw: Supporting Investigative Analysis through Interactive Visualization, Information Visualization, vol. 7, no. 2, pp. 118–132, Jun. 2008, doi: 10.1057/palgrave.ivs.9500180. | doi.org/10.1057/palgrave.ivs.9500180 | System | Presents a visual analytic system that represents documents and their entities visually to increase analytic efficiency | Based on Pirolli and Card, specifically two pain points they identified in the analytic pross: 1. cost of identifying items for further attention and 2. attention span for evidence and hypotheses Uses relational lists and graphs (looks knowledge graph-esque) as well as NER text view (other views as well, but these are most relevant to TLDR) | N/A | RQ2-Process | System designed to aid analytic process | Development of an interactive and dynamic system to assist analyts with foraging and sense-making activities across collections of textual reports | No evaluation, justification for design is primarily based on Pirolli and Card, which may not be widely representative | Tools/Systems | 46 | |||||
41 | The Sandbox for analysis: concepts and methods | Wright, William,"Schroh, David","Proulx, Pascale","Skaburskis, Alex","Cort, Brian" | 2006 | The Sandbox is a flexible and expressive thinking environment that supports both ad-hoc and more formal analytical tasks. It is the evidence marshalling and sensemaking component for the analytical software environment called nSpace. This paper presents innovative Sandbox human information interaction capabilities and the rationale underlying them including direct observations of analysis work as well as structured interviews. Key capabilities for the Sandbox include “put-this-there” cognition, automatic process model templates, gestures for the fluid expression of thought, assertions with evidence and scalability mechanisms to support larger analysis tasks. The Sandbox integrates advanced computational linguistic functions using a Web Services interface and protocol. An independent third party evaluation experiment with the Sandbox has been completed. The experiment showed that analyst subjects using the Sandbox did higher quality analysis in less time than with standard tools. Usability test results indicated the analysts became proficient in using the Sandbox with three hours of training. | W. Wright, D. Schroh, P. Proulx, A. Skaburskis, and B. Cort, The Sandbox for analysis: concepts and methods, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Montréal Québec Canada, Apr. 2006, pp. 801–810. doi: 10.1145/1124772.1124890. | https://dl.acm.org/doi/10.1145/1124772.1124890 | System | Evidence marshalling and sense-making component called the Sandbox for an analytical software environment | Key capabilities for the Sandbox include “put-this-there” cognition, automatic process model templates, gestures for the fluid expression of thought, assertions with evidence and scalability mechanisms to support larger analysis tasks. Thorough study with good background/grounding in three CTA studies conducted by the research team on actual intelligence analysts | Intelligence analysts | RQ2-Process | Sense-making system for analysts | Sense-making component of analytic architecture with flexible and varied note-taking abilities to facilitate variety of analytical methods | Focus is more on note-taking and keeping pertinent information in working memory than on linking/sorting large amounts of potentially relevant and/or related information. Evaluation was small/preliminary and results were self-reported (and compared effectiveness of system w effectiveness of MS Word) | Tools/Systems | 47 | |||||
42 | Stories in GeoTime | Eccles, Ryan,"Kapler, Thomas","Harper, Robert","Wright, William" | 2008 | A story is a powerful abstraction used by intelligence analysts to conceptualize threats and understand patterns as part of the analytical process. This paper demonstrates a system that detects geo-temporal patterns and integrates story narration to increase analytic sense-making cohesion in GeoTime. The GeoTime geo-temporal event visualization tool was augmented with a story system that uses narratives, hypertext-linked visualizations, visual annotations, and pattern detection to create an environment for analytic exploration and communication, thereby assisting the analyst in identifying, extracting, arranging, and presenting stories within the data. The story system lets analysts operate at the story level with higher level abstractions of data, such as behaviors and events, while staying connected to the evidence. The story system was developed in collaboration with analysts. A formal evaluation was completed that showed high utility and usability. | R. Eccles, T. Kapler, R. Harper, and W. Wright, Stories in GeoTime, Information Visualization, vol. 7, no. 1, pp. 3–17, Mar. 2008, doi: 10.1057/palgrave.ivs.9500173. | doi.org/10.1057/palgrave.ivs.9500173 | System | System designed to enable analysts to perform analysis at a higher level of abstraction than currently possible | Intelligence analysts | RQ2-Process | Narrative based pattern identification system for analysts | System that uses narrative story structure to aid analysis through identification of geo-temporal patterns | There's a model in here but it is just terrible | Tools/Systems | 48 | ||||||
43 | Curing Analytic Pathologies: Pathways to Improved Intelligence Analysis | Cooper, Jeffrey R. | 2005 | As a result of a number of analytic projects for different intelligence agencies, a major focus of my work during the past several years has involved examining the practice of analysis within the US Intelligence Community. This study was prompted by a growing conviction-shared by others, to be sure-that improving the analytic products delivered by Intelligence Community components had to begin with a critical and thorough appraisal of the way those products are created. A conversation with a physicist friend in 2002 had triggered thoughts on several basic differences between the practice of science and intelligence analysis. Shortly thereafter, an invitation to give a seminar on intelligence analysis at Stanford University led me to prepare a briefing entitled Intelligence and Warning Analytic Pathologies, which focused on a diagnosis of the problems highlighted by recent intelligence failures. As Donald Stokes noted in his seminal book on science and technological innovation, Pasteurs Quadrant, Pathologies have proved to be both a continuing source of insight into the systems normal functioning and a motive for extending basic knowledge. The Analytic Pathologies framework yields four insights that are crucial both to accurate diagnosis and to developing effective remedies. First, the framework enables analysts to identify individual analytic impediments and determine their sources. Second, it prompts analysts to detect the systemic pathologies that result from closely-coupled networks and to find the linkages among the individual impediments. Third, it demonstrates that each of these networks, and thus each systemic pathology, usually spans multiple levels within the hierarchy of the Intelligence Community. Fourth, the framework highlights the need to treat both the systemic pathologies and the individual impediments by focusing effective remedial measures on the right target and at the appropriate level. | J. R. Cooper, Curing Analytic Pathologies: Pathways to Improved Intelligence Analysis, CENTRAL INTELLIGENCE AGENCY WASHINGTON DC CENTER FOR STUDY OF INTELLIGENCE, Dec. 2005. Accessed: Jun. 29, 2022. [Online]. Available at: https://apps.dtic.mil/sti/citations/ADA500058 | https://apps.dtic.mil/sti/citations/ADA500058 | Theory/Model | Analytic Pathologies framework uses pathology as a metaphor for analysis to improve analytic products by better understanding analytic process | RQ3-Job | Assesses failures of the intelligence community at large and provides recommendations for improvement at the analyst level and full community level | Recommendations for how to avoid intelligence failures | 49 | |||||||||
44 | The CACHE Study: Group Effects in Computer-supported Collaborative Analysis | Convertino, Gregorio,"Billman, Dorrit","Pirolli, Peter","Massar, J. P.","Shrager, Jeff" | 2008 | The present experiment investigates effects of group composition in computer-supported collaborative intelligence analysis. Human cognition, though highly adaptive, is also quite limited, leading to systematic errors and limitations in performance – that is, biases. We experimentally investigated the impact of group composition on an individual’s bias, by composing groups that differ in whether their members initial beliefs are diverse (heterogeneous group) or similar (homogeneous group). We study three-member, distributed, computer-supported teams in heterogeneous, homogeneous, and solo (or nominal) groups. We measured bias in final judgment, and also in the selection and evaluation of the evidence that contributed to the final beliefs. The distributed teams collaborated via CACHE-A, a web-based software environment that supports a collaborative version of Analysis of Competing Hypotheses (or ACH, a method used by intelligence analysts). Individuals in Heterogeneous Groups showed no net process cost, relative to noninteracting individuals. Both heterogeneous and solo (noninteracting) groups debiased strongly, given a stream of balanced evidence. In contrast, individuals in Homogenous Groups did worst, accentuating their initial bias rather than debiasing. We offer suggestions about how CACHE-A supports collaborative analysis, and how experimental investigation in this research area can contribute to design of CSCW systems. | G. Convertino, D. Billman, P. Pirolli, J. P. Massar, and J. Shrager, The CACHE Study: Group Effects in Computer-supported Collaborative Analysis, Comput Supported Coop Work, vol. 17, no. 4, pp. 353–393, Aug. 2008, doi: 10.1007/s10606-008-9080-9. | doi.org/10.1007/s10606-008-9080-9 | Evaluation | Evaluates impact of group dynamics on reducing/increasing bias | 27 college studdents | RQ4-Collab | Shows how collaboration can be harmful to challenging biases if collaborators all have same viewpoints | "Both heterogeneous and solo (noninteracting) groups debiased strongly, given a stream of balanced evidence. In contrast, individuals in Homogenous Groups did worst, accentuating their initial bias rather than debiasing." | 50 | ||||||||
45 | Restructuring structured analytic techniques in intelligence | Chang, Welton,"Berdini, Elissabeth","Mandel, David R.","Tetlock, Philip E." | 2018 | Structured analytic techniques (SATs) are intended to improve intelligence analysis by checking the two canonical sources of error: systematic biases and random noise. Although both goals are achievable, no one knows how close the current generation of SATs comes to achieving either of them. We identify two root problems: (1) SATs treat bipolar biases as unipolar. As a result, we lack metrics for gauging possible over-shooting—and have no way of knowing when SATs that focus on suppressing one bias (e.g., overconfidence) are triggering the opposing bias (e.g., under-confidence); (2) SATs tacitly assume that problem decomposition (e.g., breaking reasoning into rows and columns of matrices corresponding to hypotheses and evidence) is a sound means of reducing noise in assessments. But no one has ever actually tested whether decomposition is adding or subtracting noise from the analytic process—and there are good reasons for suspecting that decomposition will, on balance, degrade the reliability of analytic judgment. The central shortcoming is that SATs have not been subject to sustained scientific of the sort that could reveal when they are helping or harming the cause of delivering accurate assessments of the world to the policy community. | W. Chang, E. Berdini, D. R. Mandel, and P. E. Tetlock, Restructuring structured analytic techniques in intelligence, Intelligence and National Security, vol. 33, no. 3, pp. 337–356, Apr. 2018, doi: 10.1080/02684527.2017.1400230. | doi.org/10.1080/02684527.2017.1400230 | Theory/Model | Structure analytic techniques (SATs) - methods to reduce analytic bias - may have shortcomings that overlook, or promote, bias | RQ2-Process | Descibes methods that analysts in the IC use to reduce bias in analytic process | Table of existing SATs, what cognitive biases they target, and what cognitive biases they may create. Identification of 2 large-scale issues with SATs | 51 | |||||||||
46 | Recommender systems: from algorithms to user experience | Konstan, Joseph A.,"Riedl, John" | 2012 | Since their introduction in the early 1990’s, automated recommender systems have revolutionized the marketing and delivery of commerce and content by providing personalized recommendations and predictions over a variety of large and complex product offerings. In this article, we review the key advances in collaborative filtering recommender systems, focusing on the evolution from research concentrated purely on algorithms to research concentrated on the rich set of questions around the user experience with the recommender. We show through examples that the embedding of the algorithm in the user experience dramatically affects the value to the user of the recommender. We argue that evaluating the user experience of a recommender requires a broader set of measures than have been commonly used, and suggest additional measures that have proven effective. Based on our analysis of the state of the field, we identify the most important open research problems, and outline key challenges slowing the advance of the state of the art, and in some cases limiting the relevance of research to real-world applications. | J. A. Konstan and J. Riedl, Recommender systems: from algorithms to user experience, User Model User-Adap Inter, vol. 22, no. 1–2, pp. 101–123, Apr. 2012, doi: 10.1007/s11257-011-9112-x. | 10.1007/s11257-011-9112-x | 52 | ||||||||||||||
47 | Multicriteria User Modeling in Recommender Systems | Lakiotaki, Kleanthi,"Matsatsinis, Nikolaos F.","Tsoukiàs, Alexis" | 2011 | The paper mentions that a hybrid recommender systems framework creates user-profile groups before applying a collaborative-filtering algorithm by incorporating techniques from the multiple-criteria decision-analysis (MCDA) field. | K. Lakiotaki, N. F. Matsatsinis, and A. Tsoukiàs, Multicriteria User Modeling in Recommender Systems, IEEE Intelligent Systems, vol. 26, no. 2, pp. 64–76, Mar. 2011, doi: 10.1109/MIS.2011.33. | 10.1109/MIS.2011.33 | 53 | ||||||||||||||
48 | Deconstructing Categorization in Visualization Recommendation: A Taxonomy and Comparative Study | Lee, Doris Jung-Lin,"Setlur, Vidya","Tory, Melanie","Karahalios, Karrie G.","Parameswaran, Aditya" | 2021 | Visualization recommendation (VisRec) systems provide users with suggestions for potentially interesting and useful next steps during exploratory data analysis. These recommendations are typically organized into categories based on their analytical actions, i.e., operations employed to transition from the current exploration state to a recommended visualization. However, despite the emergence of a plethora of VisRec systems in recent work, the utility of the categories employed by these systems in analytical workflows has not been systematically investigated. Our paper explores the efficacy of recommendation categories by formalizing a taxonomy of common categories and developing a system, Frontier, that implements these categories. Using Frontier, we evaluate workflow strategies adopted by users and how categories influence those strategies. Participants found recommendations that add attributes to enhance the current visualization and recommendations that filter to sub-populations to be comparatively most useful during data exploration. Our findings pave the way for next-generation VisRec systems that are adaptive and personalized via carefully chosen, effective recommendation categories. | D. J.-L. Lee, V. Setlur, M. Tory, K. G. Karahalios, and A. Parameswaran, Deconstructing Categorization in Visualization Recommendation: A Taxonomy and Comparative Study, IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2021, doi: 10.1109/TVCG.2021.3085751. | 10.1109/TVCG.2021.3085751 | 54 | ||||||||||||||
49 | Making recommendations better: an analytic model for human-recommender interaction | McNee, Sean M.,"Riedl, John","Konstan, Joseph A." | 2006 | Recommender systems do not always generate good recommendations for users. In order to improve recommender quality, we argue that recommenders need a deeper understanding of users and their information seeking tasks. Human-Recommender Interaction (HRI) provides a framework and a methodology for understanding users, their tasks, and recommender algorithms using a common language. Further, by using an analytic process model, HRI becomes not only descriptive, but also constructive. It can help with the design and structure of a recommender system, and it can act as a bridge between user information seeking tasks and recommender algorithms. | S. M. McNee, J. Riedl, and J. A. Konstan, Making recommendations better: an analytic model for human-recommender interaction, in CHI ’06 Extended Abstracts on Human Factors in Computing Systems, Montréal Québec Canada, Apr. 2006, pp. 1103–1108. doi: 10.1145/1125451.1125660. | https://dl.acm.org/doi/10.1145/1125451.1125660 | 55 | ||||||||||||||
50 | Evaluating recommender systems from the user’s perspective: survey of the state of the art | Pu, Pearl,"Chen, Li","Hu, Rong" | 2012 | P. Pu, L. Chen, and R. Hu, Evaluating recommender systems from the user’s perspective: survey of the state of the art, User Model User-Adap Inter, vol. 22, no. 4–5, pp. 317–355, Oct. 2012, doi: 10.1007/s11257-011-9115-7. | 10.1007/s11257-011-9115-7 | 56 | |||||||||||||||
51 | Evaluating Recommendation Systems | Shani, Guy,"Gunawardana, Asela" | 2011 | Recommender systems are now popular both commercially and in the research community, where many approaches have been suggested for providing recommendations. In many cases a system designer that wishes to employ a recommendation system must choose between a set of candidate approaches. A first step towards selecting an appropriate algorithm is to decide which properties of the application to focus upon when making this choice. Indeed, recommendation systems have a variety of properties that may affect user experience, such as accuracy, robustness, scalability, and so forth. In this paper we discuss how to compare recommenders based on a set of properties that are relevant for the application. We focus on comparative studies, where a few algorithms are compared using some evaluation metric, rather than absolute benchmarking of algorithms. We describe experimental settings appropriate for making choices between algorithms. We review three types of experiments, starting with an offline setting, where recommendation approaches are compared without user interaction, then reviewing user studies, where a small group of subjects experiment with the system and report on the experience, and finally describe large scale online experiments, where real user populations interact with the system. In each of these cases we describe types of questions that can be answered, and suggest protocols for experimentation. We also discuss how to draw trustworthy conclusions from the conducted experiments. We then review a large set of properties, and explain how to evaluate systems given relevant properties. We also survey a large set of evaluation metrics in the context of the property that they evaluate. | G. Shani and A. Gunawardana, Evaluating Recommendation Systems, in Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. B. Kantor, Eds. Boston, MA: Springer US, 2011, pp. 257–297. doi: 10.1007/978-0-387-85820-3_8. | http://link.springer.com/10.1007/978-0-387-85820-3_8 | 57 | ||||||||||||||
52 | Evaluating the effectiveness of explanations for recommender systems: Methodological issues and empirical studies on the impact of personalization | Tintarev, Nava,"Masthoff, Judith" | 2012 | When recommender systems present items, these can be accompanied by explanatory information. Such explanations can serve seven aims: effectiveness, satisfaction, transparency, scrutability, trust, persuasiveness, and efficiency. These aims can be incompatible, so any evaluation needs to state which aim is being investigated and use appropriate metrics. This paper focuses particularly on effectiveness (helping users to make good decisions) and its trade-off with satisfaction. It provides an overview of existing work on evaluating effectiveness and the metrics used. It also highlights the limitations of the existing effectiveness metrics, in particular the effects of under- and overestimation and recommendation domain. In addition to this methodological contribution, the paper presents four empirical studies in two domains: movies and cameras. These studies investigate the impact of personalizing simple feature-based explanations on effectiveness and satisfaction. Both approximated and real effectiveness is investigated. Contrary to expectation, personalization was detrimental to effectiveness, though it may improve user satisfaction. The studies also highlighted the importance of considering opt-out rates and the underlying rating distribution when evaluating effectiveness. | N. Tintarev and J. Masthoff, Evaluating the effectiveness of explanations for recommender systems: Methodological issues and empirical studies on the impact of personalization, User Model User-Adap Inter, vol. 22, no. 4–5, pp. 399–439, Oct. 2012, doi: 10.1007/s11257-011-9117-5. | 10.1007/s11257-011-9117-5 | 58 | ||||||||||||||
53 | Storytelling in entity networks to support intelligence analysts | "Hossain, M. Shahriar", "Butler, Patrick", "Boedihardjo, Arnold P.", "Ramakrishnan, Naren" | 2012 | Intelligence analysts grapple with many challenges, chief among them is the need for software support in storytelling, i.e., automatically 'connecting the dots' between disparate entities (e.g., people, organizations) in an effort to form hypotheses and suggest non-obvious relationships. We present a system to automatically construct stories in entity networks that can help form directed chains of relationships, with support for co-referencing, evidence marshaling, and imposing syntactic constraints on the story generation process. A novel optimization technique based on concept lattice mining enables us to rapidly construct stories on massive datasets. Using several public domain datasets, we illustrate how our approach overcomes many limitations of current systems and enables the analyst to efficiently narrow down to hypotheses of interest and reason about alternative explanations. | https://dl.acm.org/doi/abs/10.1145/2339530.2339742 | Technique | 59 | ||||||||||||||
54 | Analyzing evolving stories in news articles | "Camacho Barranco, Roberto", "Boedihardjo, Arnold P", "Hossain, M Shahriar" | 2019 | There is an overwhelming number of news articles published every day around the globe. Following the evolution of a news story is a difficult task given that there is no such mechanism available to track back in time to discover and study the hidden relationships between relevant events in digital news feeds. The techniques developed so far to extract meaningful information from a massive corpus rely on similarity search, which results in a myopic loopback to the same topic without providing the needed insights to hypothesize the origin of a story that may be completely different than the news today. In this paper, we present an algorithm that mines historical news data to detect the origin of an event, segments the timeline into disjoint groups of coherent news articles, and outlines the most important documents in a timeline with a soft probability to provide a better understanding of the evolution of a story. Qualitative and quantitative evaluations of our framework demonstrate that our algorithm discovers statistically significant and meaningful stories in reasonable time. Additionally, a relevant case study on a set of news articles demonstrates that the generated output of the algorithm holds the promise to aid prediction of future entities (e.g., actors) in a story. | https://link.springer.com/article/10.1007/s41060-017-0091-9 | Technique | 60 | ||||||||||||||
55 | Human-Centered Study of a Network Operations Center: Experience Report and Lessons Learned | Celeste Lyn Paul | 2014 | Network operations centers are notoriously difficult places to conduct human-centered research. The intense pace and sensitive information environment creates a number of hurdles for researchers. This paper shares the experiences from humancentered research of a government network operations center. The lessons learned from conducting interviews, field observations, and a card sorting study offer guidance to those who may study network operations centers in the future | Paul, Celeste Lyn. "Human-centered study of a network operations center: experience report and lessons learned." Proceedings of the 2014 ACM Workshop on Security Information Workers. 2014. | https://dl.acm.org/doi/pdf/10.1145/2663887.2663899 | Conducted interviews with 7 people who had experience working with or in the operations center prior to visiting the operations center. Observed operations center over a 12 month period. A card sorting method was then used to explore mental models of cyber situational awareness in the operations center. | Findings include: - Operations center is highly collaborative, however most information is shared verbally or through physically co-located interactions, which does not support knowledge transfer or produce many artifacts - For shift managers the biggest challenge is maintaining "mission-level" situational awareness and this would be where greatest improvements could be made through new tools or artifacts | Interviews with 7 people (all male), not all analysts in the operations center; 30 hours of observation over 12 months; Card sorting activity with 12 analysts and managers (all male) | RQ2-Process | The findings discuss some relevant aspects of how analysts work in a network operations center. | Human- centered research of a government network operations center | Half of the interviews were not with actual analysts. Observations were infrequent. Doesan't actual describe the analyst process or information used. | 61 | |||||||
56 | Improving Interpretability for Cyber Vulnerability Assessment Using Focus and Context Visualizations | Alperin, Kenneth B.,"Wollaber, Allan B.","Gomez, Steven R." | 2020 | Risk scoring provides a simple and quantifiable metric for decision support in cyber security operations, including prioritizing how to address discovered software vulnerabilities. However, scoring systems are often opaque to operators, which makes scores difficult to interpret in the context of their own networks, each other, or in a broader threat landscape. This interpretability challenge is exacerbated by recent applications of artificial intelligence (AI) and machine learning (ML) for vulnerability assessment, where opaque machine reasoning can hinder domain experts’ trust in the decision-support toolkit or the actionability of its outputs. In this paper, we address this challenge through a combination of visualizations and analytics that complement existing techniques for vulnerability assessment. We present a study toward designing more interpretable visual encodings for decision support for vulnerability assessment. In particular, we consider the problem of making datasets of known vulnerabilities more interpretable at multiple scales, inspired by focus and context principles from the information visualization design community. The first scale considers individually scored vulnerabilities by using an explainable AI (XAI) toolkit for an ML risk-scoring model and by developing new visualizations of CVSS score features. The second scale uses an embedding for vulnerability descriptions to cluster potentially similar vulnerabilities. We outline use cases for these tools and discuss opportunities for applying XAI concepts to cyber risk and vulnerability management. | K. B. Alperin, A. B. Wollaber, and S. R. Gomez, Improving Interpretability for Cyber Vulnerability Assessment Using Focus and Context Visualizations, in 2020 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2020, pp. 30–39. doi: 10.1109/VizSec51108.2020.00011. | doi.org/10.1109/VizSec51108.2020.00011 | System | Recommends XAI with the prioritorizing of issues to address for cyber analysts | RQ2-Process | RQ1-Info | Describes how XAI should be applied to defining the decision support tools of an analyst | 64 | |||||||||
57 | Developing a core ontology to improve military intelligence analysis | Valentina Dragos | 2013 | In highly dynamic and heterogeneous environments, providing commanders with decision making support requires a through understanding of processes involved and the development of underlying knowledge models upon which reasoning mechanisms can be based. This paper presents the construction of ONTO-CIF, a formal ontology created to improve intelligence analysis. ONTO-CIF was developed by following a methodology based on textual documents, which allows us to accomplish a satisfactory accuracy level in terms of domain coverage while remaining on a manageable scale size. The paper also illustrates several semantic-based scenarios to support intelligence analysis, a central task of the military application field. | Dragos, V. (2013). Developing a core ontology to improve military intelligence analysis. International Journal of Knowledge-based and Intelligent Engineering Systems, 17(1), 29-36. | https://web.s.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=8b5c5dff-3cca-41a4-8077-4b79ded5b68c%40redis | Theory/Model | Proposes an ontology to categorize variables used in intelligence analysis | RQ1-Info | 65 | |||||||||||
58 | Visualizing cyber security: Usable workspaces | Fink, Glenn A.,"North, Christopher L.","Endert, Alex","Rose, Stuart" | 2009 | The goal of cyber security visualization is to help analysts increase the safety and soundness of our digital infrastructures by providing effective tools and workspaces. Visualization researchers must make visual tools more usable and compelling than the text-based tools that currently dominate cyber analysts’ tool chests. A cyber analytics work environment should enable multiple, simultaneous investigations and information foraging, as well as provide a solution space for organizing data. We describe our study of cyber-security professionals and visualizations in a large, high-resolution display work environment and the analytic tasks this environment can support. We articulate a set of design principles for usable cyber analytic workspaces that our studies have brought to light. Finally, we present prototypes designed to meet our guidelines and a usability evaluation of the environment. | G. A. Fink, C. L. North, A. Endert, and S. Rose, Visualizing cyber security: Usable workspaces, in 2009 6th International Workshop on Visualization for Cyber Security, Oct. 2009, pp. 45–56. doi: 10.1109/VIZSEC.2009.5375542. | doi.org/10.1109/VIZSEC.2009.5375542 | Theory/Model | Describes the analytic tasks of cyber analysts with a large high-resolution display. | we observed analysts on a large, high-resolution workspace solving the VAST2009 challenge. Then, we prototyped visualizations that address the concerns we received from the interviews and con- ducted follow-up interviews to receive feedback on our designs The following solutions would greatly enhance the performance of cyber analysts: 1. A way to provide rich linkages among multiple visualiza- tion tools that better support the entire process of analysis. 2. Tools that help frame queries built from natural interac- tions with the data rather than via SQL statements. 3. A means of keeping a visual history of the manipulation steps analysts took to achieve a particular representation. 4. Input devices, controls, and window managers that work well for large displays. | Handful of Cyber Analysts | RQ2-Process | Provided a design prototype of multiple screens and observed how this impacted users. Designed a set of prototype tools that work better with multiple screens and then interviewed users. Conducted an ergonomics study to better undersand the long-term effects of the multi-screen set up | Design Prototypes and usability evaluations for analysis workspaces | 66 | |||||||
59 | A Taxonomy of Cyber Awareness Questions for the User-Centered Design of Cyber Situational Awareness | Celeste Lyn Paul and Kirsten Whitley | 2013 | This paper offers insights to how cyber security analysts establish and maintain situation awareness of a large computer network. Through a series of interviews, observations, and a card sorting activity, we examined the questions analysts asked themselves during a network event. We present the results of our work as a taxonomy of cyber awareness questions that represents a mental model of situation awareness in cyber security analysts. | Paul, Celeste Lyn, and Kirsten Whitley. "A taxonomy of cyber awareness questions for the user-centered design of cyber situation awareness." International conference on human aspects of information security, privacy, and trust. Springer, Berlin, Heidelberg, 2013 | https://link.springer.com/content/pdf/10.1007/978-3-642-39345-7_16.pdf | Theory/Model | Defines a taxonomy of cyber awareness questions that represents a mental model of situation awareness in cyber security analysts. | Based on the interviews, observations, and card sorting done in [61] | See [61] | RQ2-Process | A taxonomy of cyber awareness questions that describes a set of questions that analysts ask themselves while they establish and maintain situational awareness during a network event | Limited to cyber situational awareness (also missing intrusion detection) | 67 | |||||||
60 | Attributes of an analyst: What we can learn from the intelligence analysts job description | "Corkill, Jeffrey D", "Cunow, Teresa Kasprzyk", "Ashton, Elisabeth", "East, Amanda" | 2015 | Intelligence is a function embedded in the organisational structures of government agencies and departments at the federal, state and local level. The intelligence analyst plays an important role in supporting the operational and policy decision makers in those organisations. Notwithstanding that important role, there has been limited research into the attributes of good analysts. In the course of this research we examined 300 advertised analyst job descriptions and compared the attributes sought with those attributes described in the literature. Whilst some correlation was identified, the generic nature of the attributes sought suggests that it may be possible they have a negative influence on the quality of candidates applying for the roles. This research is significant in that it suggests that organisations and departments may need to rethink the construction of the analyst job description. | https://ro.ecu.edu.au/cgi/viewcontent.cgi?article=1041&context=asi | Review_ | Provided a qualitative analysis on various job descriptions of intelligence analysts and compared it with literature. The idea is come up with a common understanding of what the necessary attributes of an analyst are. | This is in the context of Australian analysts. | RQ3-Job | Describes the nature of the function of analysts and reduced it to two key concepts: thinking and communication. | 68 | ||||||||||
61 | An analysis of expertise in intelligence analysis to support the design of Human-Centered Artificial Intelligence | "Hepenstal, Sam", "Zhang, Leishi", "Wong, B L William" | 2021 | Intelligence analysis involves unpredictable processes and decision making about complex domains where analysts rely upon expertise. Artificial Intelligence (AI) systems could support analysts as they perform analysis tasks, to enhance their expertise. However, systems must also be cognisant about how expertise is gained and designed so that this is not impinged. In this paper, we describe the results of Cognitive Task Analysis interviews with 6 experienced intelligence analysts. We capture themes, in terms of their decision making paths during an analysis task, and highlight how each theme is both influenced by expertise and an influence upon expertise. We also identify important interdependencies between themes. We propose that our findings can be used to help design Human-Centered AI (HCAI) systems for supporting intelligence analysts. | https://ieeexplore.ieee.org/abstract/document/9659095 | Application/Design Study | Identified design considerations for an HCIA system that will support analysts in their tasks (accounting for one's expertise). | Researchers are primarily from UK but paper doesn't indicate context. Did CTA to solicit input from analysts with the goal of identifying design considerations to a proposed HCAI system that would support analysts in their tasks. | RQ2-Process | Seems to be similar to what we wanted to do. | 69 | ||||||||||
62 | Visual Decision-Support for Live Digital Forensics | Böhm, Fabian,"Englbrecht, Ludwig","Friedl, Sabrina","Pernul, Günther" | 2021 | F. Böhm, L. Englbrecht, S. Friedl, and G. Pernul, Visual Decision-Support for Live Digital Forensics, in 2021 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2021, pp. 58–67. doi: 10.1109/VizSec53666.2021.00012. | doi.org/10.1109/VizSec53666.2021.00012 | Application/Design Study | Design a Decision-Support tool for cyber analysts | RQ2-Process | Evaluates how the tool works for analysts. | 70 | |||||||||||
63 | A Visualization Interface to Improve the Transparency of Collected Personal Data on the Internet | Schufrin, Marija,"Reynolds, Steven Lamarr","Kuijper, Arjan","Kohlhammer, Jörn" | 2020 | Online services are used for all kinds of activities, like news, entertainment, publishing content or connecting with others. But information technology enables new threats to privacy by means of global mass surveillance, vast databases and fast distribution networks. Current news are full of misuses and data leakages. In most cases, users are powerless in such situations and develop an attitude of neglect for their online behaviour. On the other hand, the GDPR (General Data Protection Regulation) gives users the right to request a copy of all their personal data stored by a particular service, but the received data is hard to understand or analyze by the common internet user. This paper presents TransparencyVis - a web-based interface to support the visual and interactive exploration of data exports from different online services. With this approach, we aim at increasing the awareness of personal data stored by such online services and the effects of online behaviour. This design study provides an online accessible prototype and a best practice to unify data exports from different sources. | M. Schufrin, S. L. Reynolds, A. Kuijper, and J. Kohlhammer, A Visualization Interface to Improve the Transparency of Collected Personal Data on the Internet, in 2020 IEEE Symposium on Visualization for Cyber Security (VizSec), Oct. 2020, pp. 1–10. doi: 10.1109/VizSec51108.2020.00007. | doi.org/10.1109/VizSec51108.2020.00007 | Application/Design Study | Designs a tool for handleing the movement of personal data collected from user's online behaviors | RQ1-Info | Looks at how data moves around | Personal Data/non-collaborative | 71 | |||||||||
64 | An Application of the AKADAM Approach to Intelligence Analyst Work | Connors, Erik S.,"Craven, Patrick L.","McNeese, Michael D.","Jefferson, Tyrone","Bains, Priya","Hall, David L." | 2004 | This paper emphasizes the use of cognitive task analysis to gain significant insight into the unique domain of intelligence analysts, how intelligence analysts view this domain, and how this domain can be replicated in a controlled simulation environment in which innovative tools and procedures can be empirically tested. Details of two comprehensive knowledge elicitation sessions involving intelligence analysts are provided as an example of using the Advanced Knowledge Acquisition and Design (AKADAM) methodology to obtain contextually relevant information for use in developing a homeland defense-oriented simulation/experimental task. Several distinctive characteristics of intelligence analyst functionality were discovered, including the multi-source integration of relevant information, complex cognitive analysis, and team collaboration in decision-making. Additional themes such as social interaction and the limitations of current analysis tools were identified. | E. S. Connors, P. L. Craven, M. D. McNeese, T. Jefferson, P. Bains, and D. L. Hall, An Application of the AKADAM Approach to Intelligence Analyst Work, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 48, no. 3, pp. 627–630, Sep. 2004, doi: 10.1177/154193120404800375. | doi.org/10.1177/154193120404800375 | Technique | Uses AKADAM method of CTA to uncover characteristics of intelligence analysis | Lockheed Martin intelligence analysts (undisclosed #) | RQ1-Info | Shows concept map of information flow in IA scenario | "The present research identifies distinctive characteristics in this domain including the multi-source integration of relevant information, complex cognitive analysis, and team collaboration in decision-making." | This is mostly a methodology paper demonstrating how the AKADAM method can be used in the context of IA. It's very short and doesn't go very in depth about results/findings | Methodology | 72 | ||||||
65 | The effective analyst: a study of what makes an effective crime and intelligence analyst | Evans, Janet M.,"Kebbell, Mark R." | 2012 | The current study aimed to identify the skills and abilities required by an analyst to be recognised as effective. Thirty subject matter experts (SMEs) were engaged using the Repertory Grid Technique (RGT) to explore the specific skills and abilities of crime and intelligence analysts that result in them being deemed effective. Semi-structured interviews and the Critical Incident Technique (CIT) were conducted to strengthen these findings. Three clusters of variables were identified that indicate an effective analyst; they would have skills and abilities that could contribute to the development and dissemination of an analytical product, they would have an attitude that included being productive, seeking out work, having a high-level commitment and pride and having a ‘can do’ attitude. Some physical and personal attributes, like gender and age, were included that characterise effectiveness. The findings are discussed in relation to earlier descriptions of the skills needed to be an analyst. The findings illustrate a change from the analyst being seen as a technical specialist to a growing understanding of the analyst as part of a support structure for decision-makers. Implications of the findings for recruitment, training and development are discussed. | J. M. Evans and M. R. Kebbell, The effective analyst: a study of what makes an effective crime and intelligence analyst, Policing and Society, vol. 22, no. 2, pp. 204–219, Jun. 2012, doi: 10.1080/10439463.2011.605130. | doi.org/10.1080/10439463.2011.605130 | Theory/Model | Empirically identify characteristics that make an "effective" intelligence analyst | Context is in Australia and law enforcement - researchers used multiple methods (interviews, Reperatory Grid Technique, Critical Incident Technique) with 30 intelligence "subject matter experts" to develop a model of what makes and effective law enforcement intelligence analyst. They identify 3 primary categories - analytical product, attitudes, and attributes - that specific characteristics fit within | 30 subject matter experts (analysts, managers of analysts and end-users of analytical products working for three police services in Australia) | RQ2-Process | RQ3-Job | Highlights individual characteristics that make intelligence analysts effective (also highlighting important aspects of the work they do) | Empirically-based set of characteristics required to effectively perform the role of analyst | Quantitative analysis of qualitative data - more interested in grouping responses to highlight common ones than to dig deep on meaning. Australian law enforcement context. | Interview Study | 73 | ||||
66 | Reviewing the quality of awareness support in collaborative applications | Antunes, Pedro,"Herskovic, Valeria","Ochoa, Sergio F.","Pino, José A." | 2014 | Awareness to users is a valuable feature of a collaborative system. Therefore, the designers of a system of this type may find it useful to receive hints on the awareness support provided by the system when it is under development or evolution. This paper proposes a tool for their use to obtain suggestions on the awareness features provided by the system and those not currently supported by it. The considered kinds of awareness were obtained from a review of a significant number of proposals from the literature. The tool is based on a checklist of design elements related to these awareness types to be applied by the application designer. The construction of this checklist was done as follows. The process started with an analysis of the types of awareness to be provided. This step ended with 54 selected design elements and six awareness types. Experts on the development of collaborative systems used their experience to provide correlations between the design elements and the types of awareness previously identified, thus encapsulating their expertise within the checklist. The proposal was applied to three existing collaborative systems and the results are presented. The obtained results suggest that the checklist is adequate to provide helpful hints that may be used to improve an application’s awareness support. | P. Antunes, V. Herskovic, S. F. Ochoa, and J. A. Pino, Reviewing the quality of awareness support in collaborative applications, Journal of Systems and Software, vol. 89, pp. 146–169, Mar. 2014, doi: 10.1016/j.jss.2013.11.1078. | 10.1016/j.jss.2013.11.1078 | Theory/Model | Lays out the spaces for how collaborators maintain situational awareness, makes applicaitons and evaluates. | Identified 54 design elements froim the types of awareness represented in literature. Experts in Collaborative system designs used the design elements and types of awareness to develop checklis. The checklist/framework was applied to three systems to evaluate. Suggests hits for improving situational awareness in applications. | RQ2-Process | N/A | Looks at how people maintain awareness of work based on Literature and invited experts to help identify partterns. | 74 | ||||||||
67 | Cognitive Task Analysis: Methods to Improve Patient-Centered Medical Home Models by Understanding and Leveraging its Knowledge Work | Potworowski, Georges,"Green, Lee A." | 2013 | This brief focuses on using cognitive task analysis (CTA) to evaluate patient-centered medical home (PCMH) models. It is part of a series commissioned by the Agency for Healthcare Research and Quality (AHRQ) and developed by Mathematica Policy Research under contract, with input from other nationally recognized thought leaders in research methods and PCMH models. The series is designed to expand the toolbox of methods used to evaluate and refine PCMH models. The PCMH is a primary care approach that aims to improve quality, cost, and patient and provider experience. PCMH models emphasize patient-centered, comprehensive, coordinated, accessible care, and a systematic focus on quality and safety. | G. Potworowski and L. A. Green, Cognitive Task Analysis: Methods to Improve Patient-Centered Medical Home Models by Understanding and Leveraging its Knowledge Work, Rockville (MD), Feb. 2013, p. 14. | https://www.ahrq.gov/sites/default/files/wysiwyg/ncepcr/tools/PCMH/cognitive-task.pdf | Theory/Model | Offers some examples of Cognitive Task Analysis Processes | "macrocognitive processes include how individuals, teams, and organizations make decisions, make sense of events and experiences (called “sensemaking”), use and share knowledge, plan and replan, coordinate, learn, monitor their work, detect problems, manage the unknown, and adapt to changing conditions." | Healthcare workers | N/A | 75 | |||||||||
68 | Ontology for the Intelligence Analyst | Smith, Barry,"Malyuta, Tatiana","Salmen, David","Mandrick, William","Parent, Kesny","Bardhan, Shouvik","Johnson, Jamie" | 2012 | As available intelligence data and information expand in both quantity and variety, new techniques must be deployed for search and analytics. One technique involves the semantic enhancement of data through the creation of what are called ontologies or controlled vocabularies. When multiple different bodies of heterogeneous data are tagged by means of terms from common ontologies, then these data become linked together in ways that allow more effective retrieval and integration. We describe a simple case study to show how these benefits are being achieved, and we describe our strategy for developing a suite of ontologies to serve the needs of the war-fighter in the ever more complex battlespace environments of the future. | B. Smith et al., Ontology for the Intelligence Analyst, STATE UNIV OF NEW YORK AT BUFFALO NATIONAL CENTER FOR ONTOLOGICAL RESEARCH, Dec. 2012. Accessed: Jul. 01, 2022. [Online]. Available at: https://apps.dtic.mil/sti/citations/ADA591720 | https://apps.dtic.mil/sti/citations/ADA591720 | Technique | Discusses use of semantic enhancement ontology for intelligence analysts | This is a technical paper and is not grounded in human subjects research | 76 | |||||||||||
69 | How Analysts Think: Inference Making Strategies | "Wong, B L William", "Kodagoda, Neesha" | 2015 | In this paper we present early observations of how seven criminal intelligence analysts think and how they make inferences. We used the Critical Decision Method to identify the causal mechanisms of how they think and reason, i.e. how they organize, structure and assemble their information, understandings and inferences. We envisaged that this would enable us to design software to support the structuring of arguments and the evidential reasoning process. Our early observations suggest that analytic reasoning is not straight-forward, but appears chaotic and haphazard, and sometimes cyclic; and that inference making – abduction, induction and deduction – are not independent processes, but are closely intertwined. These processes interact dynamically, each producing outcomes that become anchors used by the others. | https://journals.sagepub.com/doi/abs/10.1177/1541931215591055 | Theory/Model | 77 | ||||||||||||||
70 | Team Situation Awareness: A Review of Definitions and Conceptual Models | She, Manrong,"Li, Zhizhong" | 2017 | Situation awareness (SA) has been a hot topic in the area of human factors and ergonomics. The SA in collaborative socio-technical systems, which is called team situation awareness (TSA), also draws increasing attentions. TSA is considered as a critical influencing factor in task performance. Like SA and many other psychological constructs, TSA receives numerous controversies in its definitions, conceptual models, theoretical underpinnings, etc. Based on a careful review of literature, this paper provides a summary and comparison of different TSA definitions, conceptual models and theoretical underpinnings. Several relevant but confusing terms are distinguished. The major controversies on TSA, including the critiques and responses, are also reviewed. The review is expected to help readers to have a comprehensive and up-to-date understanding of TSA. | M. She and Z. Li, Team Situation Awareness: A Review of Definitions and Conceptual Models, in Engineering Psychology and Cognitive Ergonomics: Performance, Emotion and Situation Awareness, Cham, 2017, pp. 406–415. doi: 10.1007/978-3-319-58472-0_31. | doi.org/10.1007/978-3-319-58472-0_31 | Theory/Model | Reviews Thories of Situational Awareness and offeres definitions to distinguish and characterize the differences | 78 | ||||||||||||
71 | From ethnography to the EAST method: A tractable approach for representing distributed cognition in Air Traffic Control | Walker, Guy H.,"Stanton, Neville A.","Baber, Chris","Wells, Linda","Gibson, Huw","Salmon, Paul","Jenkins, Daniel" | 2010 | Command and control is a generic activity involving the exercise of authority over assigned resources, combined with planning, coordinating and controlling how those resources are used. The challenge for understanding this type of activity is that it is not often amenable to the conventional experimental/methodological approach. Command and control tends to be multi-faceted (so requires more than one method), is made up of interacting socio and technical elements (so requires a systemic approach) and exhibits aggregate behaviours that emerge from these interactions (so requires methods that go beyond reductionism). In these circumstances a distributed cognition approach is highly appropriate yet the existing ethnographic methods make it difficult to apply and, for non-specialist audiences, sometimes difficult to meaningfully interpret. The Event Analysis for Systemic Teamwork method is put forward as a means of working from a distributed cognition perspective but in a way that goes beyond ethnography. A worked example from Air Traffic Control is used to illustrate how the language of social science can be translated into the language of systems analysis. Statement of Relevance: Distributed cognition provides a highly appropriate conceptual response to complex work settings such as Air Traffic Control. This paper deals with how to realise those benefits in practice without recourse to problematic ethnographic techniques. | G. H. Walker et al., From ethnography to the EAST method: A tractable approach for representing distributed cognition in Air Traffic Control, Ergonomics, vol. 53, no. 2, pp. 184–197, Feb. 2010, doi: 10.1080/00140130903171672. | doi.org/10.1080/00140130903171672 | Theory/Model | Describes how social network annalysis can be ised to characterize the interplay of communication and data shareing accross entities | No Generic model, instead describe Air Traffic Control Domain specifically | 79 | |||||||||||
72 | How Analysts Think (?): Early Observations | 2014 | https://ieeexplore.ieee.org/abstract/document/6975596 | Theory/Model | 80 | ||||||||||||||||
73 | How Analysts Think: Intuition, Leap of Faith and Insight | 2016 | https://journals.sagepub.com/doi/abs/10.1177/1541931213601039 | Theory/Model | An extension to a short paper from a different conference. | 81 | |||||||||||||||
74 | Apples to Apples: A Taxonomy of Networks in Public Management and Policy | Nowell, Branda,"Milward, H. Brinton" | 2022 | Cambridge Core - Organisation Studies - Apples to Apples | B. Nowell and H. B. Milward, Apples to Apples: A Taxonomy of Networks in Public Management and Policy, Elements in Public and Nonprofit Administration, Jun. 2022, doi: 10.1017/9781108987646. | https://www.cambridge.org/core/elements/apples-to-apples/5F0CCD49D9528EEE5C9CA6C65DEFEAD0 | Theory/Model | Describes how any social organization can be defined in terms of Structure-, System- or more relevant Purpose-Oriented networks. | Describe three perspectives on how to structure networks of actors: Purpose-oriented networks are comprised of actors who affiliate around some common purpose. By self-actualizing, purpose-oriented networks: (1) form network-level goals, (2) create an overarching system of governance, and (3) define network agency and represention within the broader environment. | We care about representing how data moves around a system of analysts. Networks emphasising purpose may be appropriate | 82 | ||||||||||
75 | Intelligence analysis ontology for cognitive assistants | Boicu, Mihai,"Tecuci, Gheorghe","Schum, David" | 2008 | This paper presents results on developing a general intelligence analysis ontology which is part of the knowledge base of Disciple-LTA, a unique and complex cognitive assistant for evidence-based hypothesis analysis that helps an intelligence analyst cope with many of the complexities of intelligence analysis. It introduces the cognitive assistant and overviews the various roles and the main components of the ontology: an ontology of “substance-blind” classes of items of evidence, an ontology of believability analysis credentials, and an ontology of actions involved in the chains of custody of the items of evidence. | M. Boicu, G. Tecuci, and D. Schum, Intelligence analysis ontology for cognitive assistants, in Proceedings of the Third International Ontology for the Intelligence Community Conference (OIC-2008), 2008, pp. 31–35. [Online]. Available at: http://ceur-ws.org/Vol-440/Proceedings.pdf#page=31 | http://ceur-ws.org/Vol-440/Proceedings.pdf#page=31 | Application/Design Study | A intelligent Agent that learns from analysts to help support them | The agent works by reducing problems into smaller and smaller problems to aid in the automatic decision support. It makes the hierarchy of evidence interactive and helps to explain the hypothesis. Knowledge base is a structured ontlogy paying attention to the relevance and believability of evidence with fine tuning for different domains of analysis. "Disciple-LTA can be used to helps new intelligence analysts learn the reasoning processes involved in making intelligence judgments and solving intelligence analysis problems." | Old Builds a tool for helping analysts understand the believeability of a source | Old, only supporting one analyst, No evaluation No way of informing the user about a change of interest. | 83 | |||||||||
76 | A data-frame theory of sensemaking | Klein, Gary,"Phillips, Jennifer K.","Rall, Erica L.","Peluso, Deborah A." | 2007 | Sensemaking is the deliberate effort to understand events. It serves a variety of functions, such as explaining anomalies, anticipating difficulties, detecting problems, guiding information search, and taking effective action. We present a data-frame theory of the process of sensemaking in natural settings. The theory contains a number of assertions. First, the theory posits that the interaction between the data and the frame is a central feature of sensemaking. The data, along with the goals, expertise, and stance of the sensemaker, combine to generate a relevant frame. The frame subsequently shapes which data from the environment will be recognized as pertinent, how the data will be interpreted, and what role they will play when incorporated into the evolving frame. Second, our observations in several domains suggest that people select frames based on a small number of anchors--highly salient data elements. Third, our research is consistent with prior work showing that expert-novice differences in sensemaking performance are not due to superior reasoning on the part of the expert or mastery of advanced reasoning strategies, but rather to the quality of the frame that is brought to bear. Fourth, we suggest that people more often construct JIT mental models from available knowledge than draw on comprehensive mental models. Fifth, the data-frame account of sensemaking is different from an information-processing description of generating inferences on data elements. People do not merely churn out inferences. They are actively trying to experience a match, however fleeting, between data and frame. (PsycInfo Database Record (c) 2020 APA, all rights reserved) | G. Klein, J. K. Phillips, E. L. Rall, and D. A. Peluso, A data-frame theory of sensemaking, in Expertise out of Context: Proceedings of the Sixth International Conference on Naturalistic Decision Making, Mahwah, NJ, US, Jan. 2007, pp. 113–155. | https://www.researchgate.net/publication/303171216_A_data-frame_theory_of_sensemaking | Theory/Model | Alternative Sensemaking model for how experts make decisions from their frame of reference. | Walks through 9 assertions that define the Data-frame Theory of Sensemaking The gist is that we work from frames, and continually reframe our perspectives to better fit the evidence | RQ2-Process | How people intentialally process information | 84 | |||||||||
77 | Towards an Ontology for Intelligence Analysis and Collection Management | Roberto Desimone and David Charles | 2002 | This short paper discusses research within the "Intelligence Support to Commanders" project as part of the UK MoD Applied Research Programme. It presents preliminary results in exploring medium/long-term concepts for the application of knowledge systems technology for intelligence support activities. An initial ontology is briefly described for intelligence analysis and collection management. The research is predominantly aimed at joint operations, but also addresses coalition issues. | https://apps.dtic.mil/sti/pdfs/ADP012334.pdf | 85 | |||||||||||||||
78 | Finding Decision Support Requirements for Effective Intelligence Analysis Tools | 2005 | https://journals.sagepub.com/doi/abs/10.1177/154193120504900318 | Theory/Model | 86 | ||||||||||||||||
79 | JIGSAW – Joint Intelligence Graphical Situation Awareness Web for Collaborative Intelligence Analysis | Smallman, H. S. | 2008 | This chapter discusses Joint Intelligence Graphical Situation Awareness Web (JIGSAW) that is designed to solve systemic problems. The intelligence cycle is in ubiquitous use across different agencies, for problems of different scope and timeline. The cycle is essentially a service model, with the operational commanders as consumers who are served products by the intelligence community in response to the requests that they generate. JIGSAW provides an Analysis Plan Generator (APG) to help analysts interpret an requests for information (RFI). An analyst uses the APG to break down an RFI to demonstrate their understanding of it, the requirements of an answer and the hypotheses that they need to test. The RFI manager supports receiving, logging, assigning, tracking and filtering the RFIs/requests that the Operational Intelligence Cell is working on. The manager is somewhat akin to a sortable MS Excel table that is used to track the progress of RFIs through the system. | https://www.taylorfrancis.com/chapters/edit/10.1201/9781315593166-18/jigsaw-joint-intelligence-graphical-situation-awareness-web-collaborative-intelligence-analysis-harvey-smallman?context=ubx&refId=ebdad06c-0993-4222-99ce-a5fc43060b28 | System | 87 | ||||||||||||||
80 | Framing and Contextualizing Information Requests: Problem Formulation as Part of the Intelligence Analysis Process | https://journals.sagepub.com/doi/abs/10.1518/155534310X12844000801087 | 88 | ||||||||||||||||||
81 | Using Information-Sharing Exchange Techniques from the Private Sector to Enhance Information Sharing Between Domestic Intelligence Organizations | Kustermann, Aaron | 2013 | Security and intelligence organizations have challenges in information sharing have resulted in incomplete information. Since 2001 state and local governments in the United States have formed information sharing hubs called fusion centers which request information from peer fusion centers as well as sharing situation reports about emerging and ongoing security situations. The requests for information (RFI) and situation awareness reporting processes are manual and occur without data standards or process standards. Public sector and private sector information sharing systems utilized both process and data standards to automate routine information sharing between organizations like those exchanges between fusion centers. These standards are coupled with information sharing tools that better enable consumer services, such as searching and booking airline travel through on-line systems, exchanges of criminal justice information using the National Information Exchange Model (NIEM) and sharing of patient and medical information utilizing the Health Information Exchange (HIE). This thesis combines the process and tools from both the public and private sector’s data and process standards and the use of information sharing tools to propose a conceptual national intelligence-sharing model (NISM). SMEs from the intelligence, counterterrorism and technology communities, within the fusion center environment, were used to review, modify, and validate NISM. | A. Kustermann (2013), Using Information-Sharing Exchange Techniques from the Private Sector to Enhance Information Sharing Between Domestic Intelligence Organizations | Dissertation/Thesis | Models information sharing for national intelligence | RQ1-Info | RQ4-Collab | 89 | |||||||||||
82 | A recognition-primed decision (RPD) model of rapid decision making | Klein | 90 | ||||||||||||||||||
83 | Information flow evaluation in autonomous groups functioning | Szczerbicki, E. | 1991 | An attempt is made to show how mathematical tools can be used in the analysis of an information flow in autonomous group functioning. The analysis is descriptive in nature and provides useful IF-THEN rules that can be used to support the process of structuring an information flow between a group and its external environment as well as information exchange within a group.<> | E. Szczerbicki, Information flow evaluation in autonomous groups functioning, IEEE Transactions on Systems, Man, and Cybernetics, vol. 21, no. 2, pp. 402–408, Mar. 1991, doi: 10.1109/21.87075. | doi.org/10.1109/21.87075 | Theory/Model | Early attempt at using mathmatics to model how teams share work with eachother (modeling information flow) | No population | summarizes the basic elements of an approach to group model development and provides mathematical tools needed to describe the information flow in group functioning | did not test their model for accuracy, just said it was in alignment with other work | 91 | |||||||||
84 | Dynamic facts in large team information sharing | Eck, Adam,"Soh, Leen-Kiat" | 2013 | In this paper, we extend the large team information sharing prob- lem to consider dynamic facts, where the value of facts about the environment being observed can change over time. Dynamic facts are challenging because the team must repeatedly converge to consistent, accurate beliefs over time, without necessarily knowing if or when the fact changes values. We discover an in- teresting, emergent phenomenon: institutional memory, where the team as a whole becomes stuck remembering outdated beliefs. We demonstrate that controlling the trust placed in new infor- mation from neighboring agents does not adequately control belief convergence with dynamic facts, which previously was shown to benefit the team when working with static facts. | A. Eck and L.-K. Soh, Dynamic facts in large team information sharing, in Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, 2013, pp. 1217–1218. [Online]. Available at: https://www.ifaamas.org/Proceedings/aamas2013/docs/p1217.pdf | https://www.ifaamas.org/Proceedings/aamas2013/docs/p1217.pdf | Theory/Model | acknowledges how challenging sending facts accross an organization is when there are many teams involved. - makes a model | call the institutional memory phenomenon: the team correctly converges its beliefs to the fact’s initial value, but then fails to properly revise its beliefs over time. Specifically, agents primarily remain stuck with their initial belief and do not even become uncertain as conflicting information is received. With respect to information flow, prior research in LTIS has primarily focused on the impact of the trust placed in new information from neigh- bors. For example, Glinton et. al [3] discovered that too little trust results in a lack of flow, whereas too much trust results in oscillat- ing beliefs as too much information is exchanged. | Describes how trust propegates through a large network and calls attention to how people do not recalibrate their trust when facts are dynamic | Simulates how information flows through a network of intelligent agents | No based on human users | 92 | ||||||||
85 | Agile Autonomous Teams in Complex Organizations | Mikalsen, Marius,"Næsje, Magne","Reime, Erik André","Solem, Anniken" | 2019 | In complex organizations, the effective functioning of autonomous teams is challenged by the need to coordinate and align work with multiple experts and other units in the organization. We report on the challenges experienced in an agile program consisting of cross-functional teams set up with resources from both the IT and business development side of the organization, while team members simultaneously remain in their line organization. Through an empirical case study of the agile program, we find that the production structure (i.e. the distribution of operational tasks) and the control structure (i.e. managing activities related to the operational task) influence agile team autonomy. We contribute by pushing past describing dependencies in terms of coordination challenges and mechanisms. To do this, we use modern sociotechnical theory to discuss how a production structure with many dependencies cause challenges and how a misaligned control structure is time-consuming and reduces team autonomy. | M. Mikalsen, M. Næsje, E. A. Reime, and A. Solem, Agile Autonomous Teams in Complex Organizations, in Agile Processes in Software Engineering and Extreme Programming – Workshops, Cham, 2019, pp. 55–63. doi: 10.1007/978-3-030-30126-2_7. | 10.1007/978-3-030-30126-2_7 | Theory/Model | Discusses with experts in IT to examine agile work environments to identify production and minutia influences have on autonomy. | 93 | ||||||||||||
86 | Algorithms that Empower? Platformization in U.S. Intelligence Analysis | Schmidt, Matthew,"Vogel, Kathleen M." | 2020 | This paper discusses a computational architecture called the Analytic Component System (ACS), which aims to provide intelligence analysts with a service-oriented computational platform. This platform is designed to empower intelligence analysts by improving the integration of people, algorithms, software, tools, and manual work in the production of time-pressured intelligence assessments. Combining the perspectives of the ACS computer science design team and an embedded social scientist, this paper will use ACS to discuss the “platformization” of intelligence analysis and what this means for how we might think about and plan for reflexive design in future computational intelligence analytic systems. | M. Schmidt and K. M. Vogel, Algorithms that Empower? Platformization in U.S. Intelligence Analysis, in 2020 IEEE International Symposium on Technology and Society (ISTAS), Nov. 2020, pp. 1–9. doi: 10.1109/ISTAS50296.2020.9555838. | doi.org/10.1109/ISTAS50296.2020.9555838 | Theory/Model | Proposes a computer architecture for intellgence analyst knowledge management and collaboration | In a prior project by LAS, they theorized and prototyped a computing architecture system called the Analytic Component System (ACS) that would centralize the analytic process by integrating information sources and providing a centralized library of tools. | N/A | RQ2-Process | RQ4-Collab | Lays out key components of analytic process and how they can be supported by computational architecture | Implications for design of intelligence analysis platform | Theory/prototype only, shut down in 2018 for unknown reasons? Makes me wonder if LAS deemed in infeasible | 94 | |||||
87 | Falling Through the Cracks: Investigation of Care Continuity in Critical Care Handoffs | Abraham, Joanna,"Almoosa, Khalid F." | 2014 | A handoff, in lay terms, refers to the act or instance of handing or transferring something to another person (to complete/to do). For instance, you hand the baton off to the next runner in a relay race. Handoffs are an everyday yet essential process in high-reliability, safety-critical settings that operate around the clock, such as between shifts at space shuttle mission controls [1, 2], nuclear power plants [3], railroad dispatch centers [4] and hospitals [5]. Regardless of the type of setting, a well-executed handoff process helps to maintain the continuity of work across shifts and between workers. | J. Abraham and K. F. Almoosa, Falling Through the Cracks: Investigation of Care Continuity in Critical Care Handoffs, in Cognitive Informatics in Health and Biomedicine: Case Studies on Critical Care, Complexity and Errors, V. L. Patel, D. R. Kaufman, and T. Cohen, Eds. London: Springer, 2014, pp. 243–269. doi: 10.1007/978-1-4471-5490-7_12. | https://doi.org/10.1007/978-1-4471-5490-7_12 | Review_ | Meta-review of Handoffs in Hospital Situations | Argues that Past work on hospital handoffs are (1) too focused on the specific moment of communication and not on the prior and post outcomes of the communication, (2) there is no clarity on what breakdowns lead to what kinds of errors, (3) and there is no gold standard for a handoff and some models are either verbose or to ambiguous. So ran an ethnographic study at a 16-bed teaching hospital in Texas with patient-care teams with clear roles. | 95 | |||||||||||
88 | User-centered design guidelines for collaborative software for intelligence analysis | "Scholtz, Jean", "Endert, Alex" | 2014 | https://ieeexplore.ieee.org/abstract/document/6867610 | 96 | ||||||||||||||||
89 | Shift Changes, Updates, and the On-Call Architecture in Space Shuttle Mission Control | Patterson, Emily S.,"Woods, David D." | 2001 | In domains such as nuclear power, industrialprocess control, and space shuttle missioncontrol, there is increased interest inreducing personnel during nominal operations. An essential element in maintaining safeoperations in high risk environments with this`on-call’ organizational architecture is tounderstand how to bring called-in practitionersup to speed quickly during escalatingsituations. Targeted field observations wereconducted to investigate what it means toupdate a supervisory controller on the statusof a continuous, anomaly-driven process in acomplex, distributed environment. Sixteenshift changes, or handovers, at the NASAJohnson Space Center were observed during theSTS-76 Space Shuttle mission. The findingsfrom this observational study highlight theimportance of prior knowledge in the updatesand demonstrate how missing updates can leaveflight controllers vulnerable to beingunprepared. Implications for mitigating riskin the transition to `on-call’ architecturesare discussed. | E. S. Patterson and D. D. Woods, Shift Changes, Updates, and the On-Call Architecture in Space Shuttle Mission Control, Computer Supported Cooperative Work (CSCW), vol. 10, no. 3, pp. 317–346, Sep. 2001, doi: 10.1023/A:1012705926828. | doi.org/10.1023/A:1012705926828 | Technique | Case study looking at how space centers hand off information during excalatioting situations and how to improve handoffs. | Analyses the 16 handoffs that took palce during a suttle mission and characterizes the topics and types of information being requested by the incoming and outgoing contollers. Based on what the controller asks it describes their prior knowledge. Aparently situational Awareness plays a part in what the incoming contoller knew to ask about. | RQ4-Collab | Talks about hand offs of infromation between teammates. Offeres methods of visualizing transcripts and variables of interest when reviewing hand off transcripts. | Analyses a handoff situation to explain how miscommunications occur | 97 | ||||||||
90 | There Is More to Monitoring a Nuclear Power Plant than Meets the Eye | Mumaw, Randall J.,"Roth, Emilie M.","Vicente, Kim J.","Burns, Catherine M." | 2000 | A fundamental challenge in studying cognitive systems in context is how to move from the specific work setting studied to a more general understanding of distributed cognitive work and how to support it. We present a series of cognitive field studies that illustrate one response to this challenge. Our focus was on how nuclear power plant (NPP) operators monitor plant state during normal operating conditions. We studied operators at two NPPs with different control room interfaces. We identified strong consistencies with respect to factors that made monitoring difficult and the strategies that operators have developed to facilitate monitoring. We found that what makes monitoring difficult is not the need to identify subtle abnormal indications against a quiescent background, but rather the need to identify and pursue relevant findings against a noisy background. Operators devised proactive strategies to make important information more salient or reduce meaningless change, create new information, and off-load some cognitive processing onto the interface. These findings emphasize the active problem-solving nature of monitoring, and highlight the use of strategies for knowledge-driven monitoring and the proactive adaptation of the interface to support monitoring. Potential applications of this research include control room design for process control and alarm systems and user interfaces for complex systems. | R. J. Mumaw, E. M. Roth, K. J. Vicente, and C. M. Burns, There Is More to Monitoring a Nuclear Power Plant than Meets the Eye, Hum Factors, vol. 42, no. 1, pp. 36–55, Mar. 2000, doi: 10.1518/001872000779656651. | doi.org/10.1518/001872000779656651 | System | Case study looking at the cognitive factors of cooperative work at nucular power plants | 98 | ||||||||||||
91 | Handoff strategies in settings with high consequences for failure: lessons for health care operations | Patterson, Emily S.,"Roth, Emilie M.","Woods, David D.","Chow, Renée","Gomes, José Orlando" | 2004 | OBJECTIVE: To describe strategies employed during handoffs in four settings with high consequences for failure. DESIGN: ANALYSIS: of observational data for evidence of use of 21 handoff strategies. SETTING: NASA Johnson Space Center in Texas, nuclear power generation plants in Canada, a railroad dispatch center in the United States, and an ambulance dispatch center in Toronto. MAIN MEASURE: Evidence of 21 handoff strategies from observations and interviews. RESULTS: Nineteen of 21 strategies were used in at least one domain, on at least an “as needed” basis. CONCLUSIONS: An understanding of how handoffs are conducted in settings with high consequences for failure can jumpstart endeavors to modify handoffs to improve patient safety. | E. S. Patterson, E. M. Roth, D. D. Woods, R. Chow, and J. O. Gomes, Handoff strategies in settings with high consequences for failure: lessons for health care operations, Int J Qual Health Care, vol. 16, no. 2, pp. 125–132, Apr. 2004, doi: 10.1093/intqhc/mzh026. | doi.org/10.1093/intqhc/mzh026 | Evaluation | Observational study in many different settings to find 21 handoff strategies | 99 | ||||||||||||
92 | Handoff Communication between Remote Healthcare Facilities | Helmig, Sara,"Cox, Jennifer","Mehta, Brinda","Burlison, Jonathan","Morgan, Jennifer","Russo, Carolyn" | 2020 | INTRODUCTION: Handoffs and transitions of care are common weak points in healthcare provider communication as patients move between sites. With no consistent pattern of communication between St. Jude Children’s Research Hospital (St. Jude) and its affiliated clinics, the Affiliate Program Office at St. Jude developed and implemented a standardized communication tool to facilitate patient transitions between different healthcare sites. METHODS: Each team of providers created flow diagrams to define the current state of communication when patients were transitioning between remote sites. Fishbone diagrams identified the common barriers to effective communication as a lack of consistent communication and ownership. We developed a communication tool to address these barriers, which was disseminated by secure email. We measured the percent usage of the completed hand-off tool before a patient transitioned, staff experience, and the number of errors. RESULTS: The time to send or receive the communication bundle was <10 minutes. Within 3 months of implementing the SMART bundle at 3 pilot sites, the bundle was used completely in 6 of 8 patient transitions and was associated with somewhat improved staff satisfaction. We identified no adverse events related to the communication bundle. CONCLUSIONS: In this small pilot study, we accomplished closed-loop communication between geographically remote healthcare sites by using an electronically transmitted standardized communication bundle. | S. Helmig, J. Cox, B. Mehta, J. Burlison, J. Morgan, and C. Russo, Handoff Communication between Remote Healthcare Facilities, Pediatr Qual Saf, vol. 5, no. 2, p. e269, Apr. 2020, doi: 10.1097/pq9.0000000000000269. | doi.org/10.1097/pq9.0000000000000269 | Technique | Developed a mnemonic for helping standardize the handoff of information between hospitals | 100 | ||||||||||||
93 | Modeling the Intelligence Analysis Process for Intelligent User Agent Development | Phillips, Joshua,"Liebowitz, Jay","Kisiel, Kenneth" | 2001 | Intelligence analysis, whether competitive intelligence, business intelligence, criminal investigation, and the like, is a critical process in many domains. Unfortunately, there have been only limited models explaining the intelligence analysis process. We have developed a comprehensive, detailed model that steps the intelligence analyst through the process of eventually arriving at a conclusion. This paper explains this model which serves as a basis for encoding the model as an intelligent user agent to be used in conjunction with the Wisdom Builder knowledge management tool. This model should greatly help intelligence analysts in building knowledge management tools to help them in their investigative work. | Phillips, J., Liebowitz, J. & Kisiel, K. (2001). Modeling the Intelligence Analysis Process for Intelligent User Agent Development, Research and Practice in Human Resource Management, 9(1), 59-73. | https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.730.697&rep=rep1&type=pdf | Theory/Model | Developed an early "detailed and comprehensive" model of the intelligence analysis process | Old paper | 102 | |||||||||||
94 | Knowledge management in the intelligence enterprise | Waltz, Edward | 2003 | E. Waltz, Knowledge management in the intelligence enterprise. Boston [Mass.]: Artech House, 2003. | http://196.190.117.157:8080/xmlui/bitstream/handle/123456789/29036/19.Edward%20Waltz.PDF?sequence=1&isAllowed=y | Book | 103 | ||||||||||||||
95 | Examining the Effects of the Value of Information on Intelligence Analyst Performance | Newcomb, E Allison | 2012 | Military intelligence analysts must deal with unprecedented amounts of data from a variety of sources. Data may originate from hard sensors, newsfeeds, video or interactions with other people. Additionally, time constraints, possibly severe consequences and dynamic, complex environments place even greater pressure on an already high pressure function. Intelligence analysts must investigate a broad range of data sources to have situational awareness. Given the abundance of data and time constraints, intelligence analysts would benefit from tools to help them quickly identify important information that is relevant in a particular context. The research discussed in this paper presents an approach for automatically presenting the valuable information first and an experimental design for evaluating decision-making performance. | E. A. Newcomb, Examining the Effects of the Value of Information on Intelligence Analyst Performance, p. 10, 2012. | http://proc.conisar.org/2012/pdf/2227.pdf | Theory/Model | Proposes an "an approach for automatically presenting the valuable information (for intelligence analysts) first and an experimental design for evaluating decision-making performance." | Differentiates quality of information and value of information which may be relevant to consider for broad TLDR goals. | RQ1-Info | RQ2-Process | Assesses quality and value of information to assist decision making process of IAs | Presents an approach and experimental design to determine the effects of making VoI weights available to intelligence analysts in varying military contexts. | Seems like it is more of a study proposal than anything else | 104 | ||||||
96 | Capturing the value of information in complex military environments: A fuzzy-based approach | Hammell, Robert J.,"Hanratty, Timothy","Heilman, Eric" | 2012 | Today’s military operations require information from an unprecedented number of sources resulting in an overwhelming volume of collected data. A primary challenge for military commanders and their staff is separating the important information from the routine. Currently, the Value of Information (VOI) assigned a piece of information is a multiple step process requiring intelligence collectors and analysts to judge its value within a host of differing operational situations. The cognitive processes behind these conclusions resist codification with exact precision suggesting that new methodologies are required to deal with this significant issue. This paper presents an approach for calculating the VOI in complex military environments using a fuzzy associative memory model as an effective framework for contextually tuning the VOI based on the information’s content, source reliability and latency. | R. J. Hammell, T. Hanratty, and E. Heilman, Capturing the value of information in complex military environments: A fuzzy-based approach, in 2012 IEEE International Conference on Fuzzy Systems, Jun. 2012, pp. 1–7. doi: 10.1109/FUZZ-IEEE.2012.6250786. | doi.org/10.1109/FUZZ-IEEE.2012.6250786 | Theory/Model | Develops a model for assessing the value of information to intelligence analysts | Uses metrics like source reliability, timeliness of information, information content and applicability to assess value of information. Intention of this is to help intelligence analysts filter out the most useful information from the overwhelming amount of information they encounter. | RQ1-Info | Models value of information | Framework for determining the value of information based on the information’s content, source reliability and latency. | Not evaluated/validated by SMEs | 105 | |||||||
97 | The impact of AI on intelligence analysis: tackling issues of collaboration, algorithmic transparency, accountability, and management | 2021 | https://www.tandfonline.com/doi/full/10.1080/02684527.2021.1946952?scroll=top&needAccess=true | Provided some design considerations to big data/AI systems. | Thinking about the intelligence analytic culture(s) and norms for how to specifically address the design, development, and use of big data/AI technologies and techniques for the IC workplace. Furthermore, factor in the organizational policies and practices first. Basically, co-design with analysts and managers. Three main issues in the paper: collaboration, transparency, and accountability (metadata). | 12 IC (de-identified) | 106 | ||||||||||||||
98 | Cognitive Predispositions and Intelligence Analyst Reasoning | Wastell, Colin A. | 2010 | C. A. Wastell, Cognitive Predispositions and Intelligence Analyst Reasoning, International Journal of Intelligence and CounterIntelligence, vol. 23, no. 3, pp. 449–460, Jun. 2010, doi: 10.1080/08850601003772802. | doi.org/10.1080/08850601003772802 | Technique | Describes types and impacts of cognitive predispositions of IAs on analysis outcomes | RQ2-Process | Emphasizes the importance of systematic analytic reasoning in the intelligence analyst process | A framework for improving intelligence analyst reasoning | I don't think this analytic reasoning framework is very relevant to our work | 107 | |||||||||
99 | Exploratory Analysis of Individuals’ Mobility Patterns and Experienced Conflicts in Workgroups | Zakaria, Camellia,"Goh, Kenneth","Lee, Youngki","Balan, Rajesh" | 2019 | Much research argues the importance of supporting social interactions in teams and communities. The field of mobile sensing alone offers significant advances in recording and understanding human and group behaviours. However, little is known about behavioural changes as a consequence of in-group phenomena. One prominent example is intra-group conflict, which naturally arises between diverse groups of people. We demonstrate the feasibility of our approach to extract mobility patterns of individual’s group behaviours sensed from a WiFi indoor localisation system and explore how these patterns relate to their team processes. 62 students enrolled in a project-intensive module, Software Engineering, were tracked over 81 days. Preliminary analysis of mobility patterns and interview data revealed differences in the mobility patterns of individuals based on their experience of conflict. | C. Zakaria, K. Goh, Y. Lee, and R. Balan, Exploratory Analysis of Individuals’ Mobility Patterns and Experienced Conflicts in Workgroups, in Proceedings of the 5th ACM Workshop on Mobile Systems for Computational Social Science, New York, NY, USA, Jun. 2019, pp. 27–31. doi: 10.1145/3325426.3329946. | https://doi.org/10.1145/3325426.3329946 | Technique | Using WIFI location indoor tracking to determin group behaviors and conflicts | Logitudinal study that looks at team dynamics from a novel perspective | Software engineering students | RQ4-Collab | Completes a logitudinal study observing how teams work and found some predictive power in where team members met and conflict | 108 | ||||||||
100 | Interactive OODA Processes for Operational Joint Human-Machine Intelligence | Leslie M Blaha | 2018 | A key advantage to strategic thinking with the Observe-Orient-Decide-Act (OODA) framework is that it provides a systematic approach to get inside the decision-making process of another agent, either cooperative or adversarial. Indeed, current OODA concepts have supported understanding human decision processes to support agile and competitive decisions about human warfighters and human-centric operations. However, future military decision making based on human-machine teaming relies on technology and interaction concepts that support joint human-machine intelligence, not just human capabilities. This requires new OODA concepts. Herein, I define a machine OODA loop, considering the characteristics that make it similar to and different from the human OODA loop. I consider how advances in artificial intelligence and cognitive modeling can be integrated within the machine-Orient stage, providing the machine a unique advantage over humans in that the machine can integrate a level of understanding and prediction about human operators together with predictions about machine behaviors and data analytics. Additionally, I propose that effective human-machine teaming should be supported by humanmachine joint decision-action processes, conceptualized as interacting OODA loops. Consideration of the interacting human-machine OODA processes offers conceptual guidance for design principles and architectures of systems supporting effective operational human-machine decision making. | Blaha, Leslie M. "Interactive OODA processes for operational joint human-machine intelligence." NATO IST-160 Specialist’s Meeting: Big Data and Military Decision Making. 2018. | https://www.sto.nato.int/publications/STO%20Meeting%20Proceedings/STO-MP-IST-160/MP-IST-160-PP-3.pdf | Theory/Model | Introduces a new conceptual machine OODA process and explores what makes the machine different and similar to human OODA processes. Also proposes ways to combine human and machine OODA processes. | None | RQ2-Process | Used the high-level conceptual framework of OODA loops to capture the decision-action processes of both humans and machines, as well as human-machine teams, in a consistent language and taxonomy | 109 |