Course Website and Github

The term no one really defined

• What is data science? 😕

Data science is the discipline of making data useful. Ok…so what is it?

The myth of data science

• Identity crisis is real in data science. And it will likely exist for a long time.
  • Data scientist
  • Applied scientist
  • Data analyst
  • Product analyst
  • Quant UX researcher: Quant UX Research at Google, We are hiring!
  • Quant researcher
  • Quant analyst
  • Business analyst
  • Business intelligence (BI) engineer
  • Analytics engineer (new to me!)
• The responsibility of a data science team is not always clear.
  • You don’t need to be in a data science team to be a data science professional.
• Many people in data science don’t do machine learning.
• Data science today is a career direction rather than a discipline. Data science is not a science!
• Focus on problem to solve and what you do rather than the title. Write your own job description!

Three tracks of data science

• Engineering (infrastructure and production): the process of making everything else possible

• Analysis: the process of turning raw information into insights in a fast way

• Modeling/Inference: the process of diving deeper into the data to discover the pattern we don’t easily see

(It is a group work from https://github.com/brohrer/academic_advisory/blob/master/authors.md !)

Engineering

1. Data environment: data storage, Kafka platform, Hadoop and Spark cluster etc.

2. Data management: parsing the logs, web scraping, API queries, and interrogating data streams.

3. Production: integrate model and analysis into the production system

Analysis

1. Domain knowledge

2. Exploratory analysis

3. Storytelling

Modeling

Problem to solve:

• Prediction/classification: image recognition, machine translation, spam/not_spam

• Explanation: customer segmentation, feature prioritization

• Causal inference: vaccine effectiveness, policy change

🔑 Questions

• Do we want to intervene?

• Is the cost of an error too high?

• Does the problem have a simple objective?

💡 Waffle Houses and Divorce Rate

• An example from Statistical Rethinking
• You can get the data here

##     Location WaffleHouses South MedianAgeMarriage Marriage Divorce
## 1    Alabama          128     1              25.3     20.2    12.7
## 2     Alaska            0     0              25.2     26.0    12.5
## 3    Arizona           18     0              25.8     20.3    10.8
## 4   Arkansas           41     1              24.3     26.4    13.5
## 5 California            0     0              26.8     19.1     8.0
## 6   Colorado           11     0              25.7     23.5    11.6

💡 Waffle Houses and Divorce Rate

💡 Waffle Houses and Divorce Rate

• Causal inference is directional but statistical association is not
• Confounding: a variable that influences both treatment and outcome causes a spurious association
• Post-treatment bias: controlling for consequence of treatment statistically knocks out treatment

Modeling

Some confusions and more to come

• Data infrastructure engineer: Go,Python, AWS/Google Cloud/Azure, logstash, Kafka, and Hadoop
• Data engineer: spark/scala, python, SQL, AWS/Google Cloud/Azure, Data modeling
• BI engineer: Tableau/looker/Mode etc, data visualization, SQL, Python
• Data analyst: SQL, basic statistics, data visualization
• Data scientist: R/Python, SQL, basic + applied statistics, data visualization, experimental design
• Research scientist: R/Python, advanced statistics + experimental design, ML, research background, publications, conference contributions, algorithms
• Applied scientist: Similar to research scientist, ML algorithm design, often with an expectation of basic software engineering skills
• Machine Learning Engineer: More advanced software engineering skillset, algorithms, machine learning algorithm design, system design
• …

Three tracks of data science

Three tracks of data science

Three tracks of data science

Data Science Types v.s Needs

Data Flow

Data Science Roles

Some links

• Recordings for this course (old versions):
  • YouTube play list
  • 2021 SMI Conference - Session 1: Introduction to Deep Learning Short Course, website
• Introduction to Data Science (reviewing): the content of the workshop is in this book. Here is the GitHub for the book with links to code notebooks.
• Awesome-Data-Science-Materials
• The Elements of Statistical Learning
• An Introduction to Statistical Learning
• The caret package
• Hard copy books:
  • Applied Predictive Modeling
  • R for Marketing Research and Analytics
  • Statistical Rethinking