Apache Spark 3 - Real-time Stream Processing using Scala

  • Course provided by Udemy
  • Study type: Online
  • Starts: Anytime
  • Price: See latest price on Udemy
Udemy

Course Description

About the Course

I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way.

Who should take this Course?

I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level.

Spark Version used in the Course

This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution.

Who this course is for:

  • Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark
  • Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark

Instructors

Architect, Author, Consultant, Trainer @ Learning Journal
  • 4.6 Instructor Rating
  • 6,233 Reviews
  • 42,449 Students
  • 10 Courses

Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects.

Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal.

He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018.

Online Training Company
  • 4.6 Instructor Rating
  • 6,233 Reviews
  • 42,449 Students
  • 10 Courses

Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry.

Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects.

Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner.

We authored various skill development courses, training, and technical articles since the beginning of the year 2018.

Expected Outcomes

  1. Real-time Stream Processing Concepts Spark Structured Streaming APIs and Architecture Working with File Streams Working With Kafka Source and Integrating Spark with Kafka State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  2. Spark Structured Streaming APIs and Architecture Working with File Streams Working With Kafka Source and Integrating Spark with Kafka State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  3. Working with File Streams Working With Kafka Source and Integrating Spark with Kafka State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  4. Working With Kafka Source and Integrating Spark with Kafka State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  5. State-less and State-full Streaming Transformations Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  6. Windowing Aggregates using Spark Stream Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  7. Watermarking and State Cleanup Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  8. Streaming Joins and Aggregation Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var input = cssToggleId && document.getElementById(cssToggleId); if (input) { if (input.dataset.type === 'checkbox') { input.dataset.checked = input.dataset.checked ? '' : 'checked'; } else { input.dataset.checked = input.dataset.allowToggle && input.dataset.checked ? '' : 'checked'; var radios = document.querySelectorAll('[name="' + input.dataset.name + '"]'); for (var i = 0; i (function(){window['__CF$cv$params']={r:'67782d480acd540f',m:'65816a7d1906dc3f6b068f2871a1bd974ee034ed-1627748666-1800-AbgZHVNzHuW4wYVhxyTzmt/7Gmn3c8kJIn+vrD4PfHRxMxNywXmmgcCbubekJqHHvrTSNGtxcs3k51GWo/GHIq6qct5J+O0Uo/InPKdQR4iUkHHte0veWM7enwlcULnsJBhGbIiYxyPr3ibuNV0x9kohxvqJcOT/d8zQOegGrfWo',s:[0x20f6b02ae2,0xe85cbd7111],}})();
  9. Handling Memory Problems with Streaming Joins Creating Arbitrary Streaming Sinks Curated for the Udemy Business collection Course content 7 sections • 33 lectures • 4h 18m total length Expand all sections Before you start 3 lectures • 4min About the Course Preview 02:58 Course Prerequisite Preview 01:19 Source Code and Other Resources 00:11 Setup your Environment 5 lectures • 33min Spark Development Environment Preview 01:44 Spark Installation Prerequisites Preview 05:25 Installing Apache Spark Preview 08:26 Setup and test your IDE 06:44 Install and run Apache Kafka 10:24 Getting started with Spark Structured Streaming 7 lectures • 1hr 11min Introduction to Stream Processing 09:16 Spark Streaming APIs - DStream Vs Structured Streaming 03:50 Creating your first stream processing application 17:40 Stream processing model in Spark 08:34 Working with Files and Directories 11:43 Streaming Sources, Sinks and Output Mode 13:05 Fault Tolerance and Restarts 06:29 Spark Streaming with Kafka 6 lectures • 45min Streaming from Kafka Source 16:19 Working with Kafka Sinks 10:32 Multi-query Streams Application 04:13 Kafka Serialization and Deserialization for Spark 05:00 Creating Kafka AVRO Sinks 04:00 Working with Kafka AVRO Source 04:52 Windowing and Aggregates 6 lectures • 1hr 2min Stateless Vs Statefull transformations 10:18 Event time and Windowing 07:14 Tumbling Window aggregate 14:05 Watermarking your windows 12:37 Watermark and output modes 09:54 Sliding Window 07:38 Stream Processing and Joins 4 lectures • 42min Joining Stream to static source 13:57 Joining Stream to another Stream 09:57 Streaming Watermark 07:17 Streaming Outer Joins 11:17 Keep Learning 2 lectures • 1min Final Word 00:50 Bonus Lecture : Get Extra 00:27 Requirements Spark Fundamentals and exposure to Spark Dataframe APIs Kafka Fundamentals and working knowledge of Apache Kafka Programming Knowledge Using Scala Programming Language A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM Description About the Course I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level. Spark Version used in the Course This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution. Who this course is for: Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark Show more Show less Instructors Prashant Kumar Pandey Architect, Author, Consultant, Trainer @ Learning Journal 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Prashant Kumar Pandey is passionate about helping people to learn and grow in their career by bridging the gap between their existing and required skills. In his quest to fulfill this mission, he is authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. With over 18 years of experience in IT as a developer, architect, consultant, trainer, and mentor, he has worked with international software services organizations on various data-centric and Bigdata projects. Prashant is a firm believer in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, he started publishing free training videos on his YouTube channel and conceptualized the idea of creating a Journal of his learning under the banner of Learning Journal. He is the founder, lead author, and chief editor of the Learning Journal portal that offers various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Learning Journal Online Training Company 4.6 Instructor Rating 6,233 Reviews 42,449 Students 10 Courses Learning Journal is a small team of people passionate about helping others learn and grow in their careers by bridging the gap between their existing and required skills. In our quest to fulfill this mission, we are authoring books, publishing technical articles, and creating training videos to help IT professionals and students succeed in the industry. Together we have over 40+ years of experience in IT as a developer, architect, consultant, trainer, and mentor. We have worked with international software services organizations on various data-centric and Bigdata projects. Learning Journal is a team of firm believers in lifelong continuous learning and skill development. To popularize the importance of lifelong continuous learning, we started publishing free training videos on our YouTube channel. We conceptualized the notion of continuous learning, creating a journal of our learning under the Learning Journal banner. We authored various skill development courses, training, and technical articles since the beginning of the year 2018. Show more Show less Udemy Business Teach on Udemy Get the app About us Contact us Careers Blog Help and Support Affiliate Impressum Kontakt Terms Privacy policy Cookie settings Sitemap © 2021 Udemy, Inc. window.handleCSSToggleButtonClick = function (event) { var target = event.currentTarget; var cssToggleId = target && target.dataset && target.dataset.cssToggleId; var in