Designing data & algorithms driven experiments

In silico experimental settings

2

Experiment

• Scientific experiment: any process in which measurements and tests are used within a (typically) controlled environment to support or refute a hypothesis.
• “evaluate the efficacy or likelihood of something previously untried”
• Hypotheses are unsupported propositions, usually based on insights or prior research, from which an investigation can be developed
• Experiments are generally designed to identify a causal relationship between variables
• Without a good experiment design, no causality can be inferred from your results

​

Experiments in data driven engineering and science: initial questions

• What is the best way to support (or disprove) our hypotheses?
• What are we measuring and how?
• What data do we need, and where can we collect it?
• What tools and libraries should we use?

These problems must be solved before initiating the experiment

Experiments in data driven engineering and science: design

• Create an experimental workflow
• Designing an experiment involves
• formulating a hypothesis or question: guides the processes necessary to then design the experiment
• determining the best way to gather the data and test the hypothesis
• identifying any problems or sources of error that may occur during the experiment
• collecting data

Data exploitation spectrum: searching, querying, analysing

6

Data

Rawness

Degree

Raw

Curated

Querying

objective

Information retrieval

Relational & aggregation querying

Data exploration

Hindsight

Insight

Foresight

Data

collections

Data

bases

Data

Querying facts

7

• Data: model of a mini-world, it is a set of facts structured according to some data model
• Query: precisely stated it can include terms, operators (and/or/negation, relational, aggregation), and constraints
• Result: collection of items that completely or partially correspond to consumers requirements (precision & recall)

8

• Average of lab & medication orders per patient and physician in a given clinic
• Vital signs and lab results used to emit a diagnosis for a given patient
• Number of patients with diabetes followed per clinic

Querying approaches

9

10

WHAT DOES CHATGPT SAY ABOUT RELATIONAL OPERATORS IN SQL?

Data Science Pipeline

Data exploration & preparation

Quantitative profiling

(descriptive statistics)

Cleaning, normalisation,

attributes engineering

Sample

selection

Insight / Foresight search

Visualization

Uni, bi, multivariable

observation

Interactive

graphs

Sample

fragmentation

Generative/Discriminative

model (training)

Validation

Assessment

• Data: observations of phenomena often described as series of features/attributes
• Query: analytics objective (looks for insights or foresights) expressed as a pipeline of operations guided by the conditions and characteristics of the data
• Result: a model or prediction with associated assessment indexes, not definitive accepted with an associated error margin, accepted by comparison

Error analysis

Ablative analysis

Raw

data

Data Science Pipeline Example

“Does it make sense to invest in low-carbon technologies?”

• Heterogeneous masses of data , often spatio-temporal and produced as streams (i.e., spatio-temporal series),
• Backbone of analytical and prediction processes

Spatio – temporal series

IoT

Data Science Pipeline Example

House &

Meterological data

“Does it make sense to invest in low-carbon technologies?”

• Exploration & preparation: Identify which variables impact my household CO₂ footprint
• Analysis: Predict whether investing in low-carbon technologies will decrease my CO₂ footprint
• Assessment: Identify which prediction model is better for my household energy consumption pattern

Series of complex and repetitive treatments and analysis

Data Science Pipeline Example

Data Science Pipeline Life Cycle

Democratized access to open data collections & algorithms

released under different conditions & qualities

7

17

18

Question & Imagine

19

Data Science Toolboxes

Enactment environments

Data Science Pipeline

• In-house programming using many different libraries, stacks, tools difficult to integrate

DATA SCIENCE STACKS

• Artisanal design depending on data scientist/engineers “expertise”

BD SERVICES PLATFORMS

DATA SCIENCE LABS

1

2

Data Science Pipeline

• Efficient execution on distributed architectures requiring important engineering effort�

• Tuning for improving data management across nodes hosting software components and libraries

1

2

CHALLENGES

DATA SCIENCE STACKS

BD SERVICES PLATFORMS

DATA SCIENCE LABS

Data divide

• Data loading
• In memory/cache/disk indexing
• Data persistence
• Query optimization
• Concurrent access
• Consistency and access control

​

​

Functions must be revisited under less strong hypothesis to support the enactment of data science pipelines

Data science pipelines’ enactment

Which tools for data science

• Programming language:
• Python one of the most flexible programming languages because it can be seen as a multiparadigm language
• Alternatives are MATLAB and R

​

• Fundamental libraries for data scientists in Python: NumPy, SciPy, Pandas & Scikit-Learn
• NumPy provides, support for multidimensional arrays with basic operations on them and useful linear algebra functions.
• SciPy provides a collection of numerical algorithms and domain-specific toolboxes, including signal processing, optimization, statistics.
• Matplotlib tools for data visualization
• Sci-kit-Learn: machine learning library with tools such as classification, regression, clustering, dimensionality reduction, model selection, and preprocessing.
• Pandas: provides high-performance data structures and data analysis tools for data manipulation with integrated indexing

​

26

Integrated Development Environment

To get started on solving data-oriented problems, we need to set up our programming environment

Decide programming language version, whether to install a data scientist ecosystem by individual tool-boxes, or to perform a bundle installation

• For example, Anaconda Python provides integration of all the Python toolboxes and applications for data scientists in a single directory

The integrated development environment (IDE) is an essential tool designed to maximize programmer productivity.

• The basic pieces of any IDE are three: the editor, the compiler, (or interpreter) and the debugger.
• Examples: PyCharm,9 WingIDE10, SPYDER (Scientific Python Development EnviRonment)

Web Integrated Development Environment

Web-based IDEs were developed considering how not only your code but also all your environment and executions can be stored in a server

​

• The server can be set up in a centre, such as a university or school
• Students can work on their homework either in the classroom or at home
• Students can execute all the previous steps over and over again, and then change some particular code cell (a segment of the document that may content source code that can be executed) and execute the operation again.

​

For example, IPython has been issued as a browser version of its interactive console: Jupyter

• Markdown (a wiki text language) cells can be added to introduce algorithms.
• It is also possible to insert Matplotlib graphics to illustrate examples or even web pages.
• Experiments can become completely and absolutely replicable.

Web Integrated Development Environment

Target Architectures

Where to design, test and deploy a data science experiment?

Hands-on