Data Science Process

 

The Data Science Process (Six Steps)

Data science projects follow a structured process. The book highlights six main steps, which help ensure that insights are reliable, reproducible, and useful for organizations.

1. Setting the Research Goal

  • What it means: Clearly define the purpose of the project.

  • How it’s done: Prepare a project charter that specifies:

    • What you’re going to research.

    • Why it benefits the organization.

    • What data and resources are needed.

    • Timeline and deliverables.

  • Example: A company may want to know: “Can we predict customer churn to improve retention?”

2. Retrieving Data

  • What it means: Collect the data required for the project.

  • Sources: Databases, spreadsheets, APIs, third-party vendors, or logs.

  • Checks needed:

    • Does the data exist?

    • Is the quality sufficient?

    • Do we have access rights?

  • Example: Gathering customer purchase records from a database or downloading open data from a government portal.

3. Data Preparation

  • What it means: Make raw data ready for analysis.

  • Sub-steps:

    • Data Cleansing: Remove errors, duplicates, or false values.

    • Data Integration: Combine data from multiple sources.

    • Data Transformation: Convert data into a usable format (e.g., encoding categories into numbers).

  • Example: If customer ages are stored differently across branches (some in years, some in birth dates), standardize them.

4. Data Exploration (EDA – Exploratory Data Analysis)

  • What it means: Gain an initial understanding of the data.

  • Techniques:

    • Descriptive statistics (mean, median, standard deviation).

    • Visualizations (histograms, scatter plots, box plots).

    • Checking distributions and relationships between variables.

  • Why: Helps detect patterns, correlations, and outliers before building models.

  • Example: Discovering that customers with low engagement (few logins) are more likely to churn.

5. Data Modeling (Model Building)

  • What it means: Apply statistical and machine learning models to answer the research question.

  • Process:

    • Select variables (features) for the model.

    • Choose modeling technique (regression, classification, clustering, etc.).

    • Train and evaluate the model iteratively.

  • Example: Using logistic regression to predict whether a customer will churn (yes/no).

6. Presentation and Automation

  • What it means: Share results and put them into use.

  • Forms: Reports, dashboards, visualizations, or presentations to stakeholders.

  • Automation: In some cases, the model needs to be automated (e.g., daily fraud detection in banking).

  • Example: A churn prediction model may be integrated into a CRM system so sales teams can act in real time.



Comments

Post a Comment

Popular posts from this blog

GNEST305 Introduction to Artificial Intelligence and Data Science KTU BTech S3 2024 Scheme - Dr Binu V P

 About Me Course Details and Syllabus GNEST305 Model Question Paper GNEST305 AI and Data Science Model Question Paper Answers GNEST305 AI and Data Science For the Implementation learn Python and R      Python for Machine Learning     R For Statistics/Data Science Module-1 Introduction to AI and Machine Learning Basics of Machine Learning Types of Machine Learning Comparison supervised,unsupervised and reinforcement learning How typical machine learning system works Challenges of Machine Learning Generalization- Bias -Variance Trade-off Supervised Learning- Linear and Logistic Regression Gradient Descent Unsupervised Learning- K Means Clustering Artificial Neural Networks - Perceptron Multilayer Perceptron and Backpropagation Universal Approximation Theorem(UAT) Regression using MLP Classification using MLP Classification Assessment Module-2 Mathematical Foundations of AI and Data Science The Role of Linear Algebra in Data Representation and Analysi...
Read more

Basics of Machine Learning

Image
Machine Learning is one of the most exciting and rapidly evolving fields in modern engineering. It allows computers to learn from data, recognize patterns, and make intelligent decisions—without being explicitly programmed for every scenario. Unlike traditional programming where we define strict rules, machine learning focuses on building systems that can adapt and improve through experience. This ability to "learn" makes it a key technology behind innovations like self-driving cars, facial recognition, speech assistants, and even advanced medical diagnosis systems. For engineering students, learning machine learning opens the door to solving real-world problems across diverse domains such as automation, smart manufacturing, predictive maintenance, and robotics. It brings together concepts from mathematics, statistics, computer science, and domain knowledge to design models that can analyze data and make accurate predictions. As the digital world generates massive amounts of...
Read more

Types of Machine Learning Systems

Image
There are so many different types of Machine Learning systems that it is useful to classify them in broad categories, based on the following criteria:      • Whether or not they are trained with human supervision ( supervised, unsupervised,                                       semisupervised, and Reinforcement Learning)      • Whether or not they can learn incrementally on the fly ( online versus batch learning)      • Whether they work by simply comparing new data points to known data points, or instead by                      detecting patterns in the training data and building a predictive model, much like scientists do                      (instance-based versus model-based learning) Machine Learning s...
Read more