Job Description
**Responsibilities**:
Design, develop, and optimize big data processing pipelines using Apache Spark and Java.
- Work on batch and real-time data processing frameworks to transform large datasets.
- Write high-performance Spark jobs using RDDs, DataFrames, and Datasets.
- Collaborate with data engineers, architects, and analysts to ensure seamless data integration.
- Optimize Spark performance through tuning, partitioning, and efficient memory management.
- Implement best practices for data governance, security, and compliance.
- Work with CI/CD pipelines, version control (Git), and automation tools for continuous deployment.
- Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
- Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
- Ensure essential procedures ar...
Design, develop, and optimize big data processing pipelines using Apache Spark and Java.
- Work on batch and real-time data processing frameworks to transform large datasets.
- Write high-performance Spark jobs using RDDs, DataFrames, and Datasets.
- Collaborate with data engineers, architects, and analysts to ensure seamless data integration.
- Optimize Spark performance through tuning, partitioning, and efficient memory management.
- Implement best practices for data governance, security, and compliance.
- Work with CI/CD pipelines, version control (Git), and automation tools for continuous deployment.
- Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
- Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
- Ensure essential procedures ar...