date

Start date

November 2022

duration

Duration

3 months

enrol

Enrol now

Book your seat

Your career in DevOps awaits you

The next batch starts in:

Program overview:

‘Data Engineering’ is a tailored but specialized course developed to assist potential Students and developers to develop their potential for Data Engineering. In this course, you’ll discover how to build an effective data architecture, streamline data processing, and maintain large-scale data systems. In addition to working with Python, you’ll also grow your language skills as you work with Shell, SQL, and other languages, to create data engineering pipelines, automate common file system tasks, and build a high-performance database.

Program overview:

‘Data Engineering’ is a tailored but specialized course developed to assist potential Students and developers to develop their potential for Data Engineering. In this course, you’ll discover how to build an effective data architecture, streamline data processing, and maintain large-scale data systems. In addition to working with Python, you’ll also grow your language skills as you work with Shell, SQL, and other languages, to create data engineering pipelines, automate common file system tasks, and build a high-performance database.

  • Students from CS, IT or Software background can join.
  • anyone who wants to learn this diverse field.
  • mprove Programming skills
  • How to deploy application on multiple clouds
  • Data Acquisition from from different platforms
Who should attend:
  • Students from CS, IT or Software background can join.
  • anyone who wants to learn this diverse field.
Objectives:
  • Improve Programming skills
  • How to deploy application on multiple clouds
  • Data Acquisition from from different platforms
Learning Outcome:
On successful completion of this program, participants will be able to:

Knowledge
  • Working knowledge of Data Engineering Ecosystem and Lifecycle.
  • Learn critical concepts around data acquisition techniques.
  • Understanding different data storage options like files, SQL and NoSQL.
  • Learn application scalability in terms of deployment.
  • Viewpoints and tips from Data professionals on starting a career in this domain.
Skills
  • Python programming basics including data structures, logic, working with files, invoking APIs, using libraries such as Pandas and Numpy, doing ETL.
  • SQL query language, SELECT, INSERT, UPDATE, DELETE statements, database functions, stored procs, working with multiple tables, JOINs, & transactions.
  • Schedule, automate, and monitor data pipelines using Apache Airflow. Run data quality checks, track data lineage, and work with data pipelines in production.
  • Orchestrate Data with Airflow.
On successful completion of this program, participants will be able to:
Knowledge
  • Working knowledge of Data Engineering Ecosystem and Lifecycle.
  • Learn critical concepts around data acquisition techniques.
  • Understanding different data storage options like files, SQL and NoSQL.
  • Learn application scalability in terms of deployment.
Skills
  • Python programming basics including data structures, logic, working with files, invoking APIs, using libraries such as Pandas and Numpy, doing ETL.
  • SQL query language, SELECT, INSERT, UPDATE, DELETE statements, database functions, stored procs, working with multiple tables, JOINs, & transactions.
  • Schedule, automate, and monitor data pipelines using Apache Airflow. Run data quality checks, track data lineage, and work with data pipelines in production.
  • Orchestrate Data with Airflow.

Become a certified Data Engineer professional
with this fast track programme

Become a certified DevOps professional with this fast track programme

Course Curriculum

Here is what's included in this DevOps Course

  • Python Basics
  • Python Data Structures
  • Working with Data in Python
  • Basics of Python + Code Best Practices using Python
  • Git (basic implementation & advance concepts)
  • Fetching & Posting data using apis (postman/python)
  • Hands-on: Python Request Library
  • Authentication for APIs
  • Oauth 2.0 flow
  • Hands-on: Fetching data from Youtube Data API
  • Hands-on: Google Sheets API
  • Any public API listed here
  • Why scrape the web?
  • How Does Web Scraping Work?
  • Techniques to scrape data from web
  • Hands-on: Web Scraping
  • Custom web Scraper (Bs4, requests)
  • Introduction to Scrapy

Shell Scripting

  • Linux & Command Line Basics
  • Linux shell: Setup
  • What is the shell?
  • Navigating, Creation and Migration commands
  • Differentiate and Pipes

Job Scheduling

  • Crontab in Linux
  • Scheduling Cron jobs: Linux
  • Scheduling Tasks with At Utility: Windows
  • Relational Database Concepts
  • Using Relational Databases
  • MySQL and PostgreSQL
  • Introduction to Cloud databases
  • Cloud Storage types and uses cases
  • Getting data from cloud database
  • Getting Started with SQL
  • Introduction to Relational Databases and Tables
  • Intermediate SQL
  • Accessing Databases using Python
  • Introduction to data pipelines
  • ELT Basics
  • Comparing ETL & ELT
  • Data Extraction Techniques
  • Introduction to Data Transformation Techniques
  • ETL using Shell Scripting
  • Batch Versus Streaming Data Pipeline Use Cases
  • Use Case: Building an Advanced ETL pipeline
  • Apache Airflow Overview
  • Advantage of Using Data Pipelines as DAGs
  • Apache Airflow UI
  • Build DAG Using Airflow
  • Airflow Monitoring and Logging
  • Hands-on: Writing DAG
  • Distributed Event Streaming Platform Components
  • Apache Kafka Overview
  • Building Event Streaming Pipelines using Kafka
  • Kafka Streaming Process
  • Data Warehouses, Data Marts, and Data Lakes
  • Designing, Modeling and Implementing Data Warehouses
  • Data Warehouse Analytics
  • Introducing NoSQL
  • Introducing MongoDB – An Open-Source NoSQL Database
  • Introducing Apache Cassandra – An Open-Source NoSQL Database
  • Hands-on: Working with NoSQL Databases
  • Introduction to Hadoop and Spark
  • introduction to MapRedude
  • Spark for Data Engineering
  • SparkML

Course Curriculum

Here is what's included in this DevOps Course

  • Python Basics
  • Python Data Structures
  • Working with Data in Python
  • Basics of Python + Code Best Practices using Python
  • Git (basic implementation & advance concepts)
  • Fetching & Posting data using apis (postman/python)
  • Hands-on: Python Request Library
  • Authentication for APIs
  • Oauth 2.0 flow
  • Hands-on: Fetching data from Youtube Data API
  • Hands-on: Google Sheets API
  • Any public API listed here
  • Why scrape the web?
  • How Does Web Scraping Work?
  • Techniques to scrape data from web
  • Hands-on: Web Scraping
  • Custom web Scraper (Bs4, requests)
  • Introduction to Scrapy

Shell Scripting

  • Linux & Command Line Basics
  • Linux shell: Setup
  • What is the shell?
  • Navigating, Creation and Migration commands
  • Differentiate and Pipes

Job Scheduling

  • Crontab in Linux
  • Scheduling Cron jobs: Linux
  • Scheduling Tasks with At Utility: Windows
  • Relational Database Concepts
  • Using Relational Databases
  • MySQL and PostgreSQL
  • Introduction to Cloud databases
  • Cloud Storage types and uses cases
  • Getting data from cloud database
  • Getting Started with SQL
  • Introduction to Relational Databases and Tables
  • Intermediate SQL
  • Accessing Databases using Python
  • Introduction to data pipelines
  • ELT Basics
  • Comparing ETL & ELT
  • Data Extraction Techniques
  • Introduction to Data Transformation Techniques
  • ETL using Shell Scripting
  • Batch Versus Streaming Data Pipeline Use Cases
  • Use Case: Building an Advanced ETL pipeline
  • Apache Airflow Overview
  • Advantage of Using Data Pipelines as DAGs
  • Apache Airflow UI
  • Build DAG Using Airflow
  • Airflow Monitoring and Logging
  • Hands-on: Writing DAG
  • Distributed Event Streaming Platform Components
  • Apache Kafka Overview
  • Building Event Streaming Pipelines using Kafka
  • Kafka Streaming Process
  • Data Warehouses, Data Marts, and Data Lakes
  • Designing, Modeling and Implementing Data Warehouses
  • Data Warehouse Analytics
  • Introducing NoSQL
  • Introducing MongoDB – An Open-Source NoSQL Database
  • Introducing Apache Cassandra – An Open-Source NoSQL Database
  • Hands-on: Working with NoSQL Databases
  • Introduction to Hadoop and Spark
  • introduction to MapRedude
  • Spark for Data Engineering
  • SparkML

Entry Requirements

There are no formal requirements for this certification.

Exams & Certification

Format: Multiple-choice
Number of questions: 40
Duration: 60 minutes Passing score: 65% (26/40)
Domains:

Meet Our Trainers

Our experienced Python Programming Language expert are here to help you understand the concepts of python programming and to apply them practically for your successful career 

Usman Sherdil

USMAN SHERDIL

Founder & CEO Sherdil IT Academy and Sherdil Cloud
BE Telecommunication & Networks

Course Materials and Support

Online Access to Course Using any Device with online method of delivery while being taught by BCS Accredited Trainer. Track Your Progress with Easy Course Navigation, Quizzes, Videos, Discussions Area.

BCS Certificate

On Successful Completion of the Exam, Receive a Professional Certificate, Accredited by BCS & Aligned to Leading IT Industry Framework - SFIA plus. Globally recognised in over 190 Countries.

BCS Membership

On Successful Completion of Exam, Receive One Free Year BCS Associate Membership (worth £92) Which includes Career Support via Springboard.

Course Materials and Support

Online Access to Course Using any Device with online method of delivery while being taught by BCS Accredited Trainer. Track Your Progress with Easy Course Navigation, Quizzes, Videos, Discussions Area.

BCS Certificate

On Successful Completion of the Exam, Receive a Professional Certificate, Accredited by BCS & Aligned to Leading IT Industry Framework - SFIA plus. Globally recognised in over 190 Countries.

BCS Membership

On Successful Completion of Exam, Receive One Free Year BCS Associate Membership (worth £92) Which includes Career Support via Springboard.

Programming languages and tools

more-questions

More Questions?

Get in touch

More Questions?

Get in touch