Data Engineer

Key Responsibilities:Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity.Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increase data accessibility, and fostering data-driven decision making across the organization.Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes.Writes unit/integration tests, contributes to engineering wiki, and documents work.Designs and develops data integrations and data quality frameworkBuilding ETL workflows, jobs or scripts or custom scripting languages to automate the data integration process.Raising CR to the Data Platform team for production deployment.Ensuring data integrity and quality within the Data Platform team. This involves setting up data validation checks, error handling mechanisms, and conducting regular audits to identify and correct data anomalies.Optimizing the Data Platform's performance to meet SLAs and end-user requirements. This may involve fine-tuning SQL queries, optimize ETL processes, or recommending hardware upgrades as needed.Providing support and assistance to end-users, including data analysts, business intelligence teams, and other stakeholders.Maintaining comprehensive documentation for the Data Platform environment, including data models, ETL processes, data dictionaries, and standard operating procedures.Collaborate with Incident management team to track and report on incidents, their resolutions, and any necessary follow-up actions.Knowledge, Skills and ExperienceCore Skills:At least 5 years of experience in the Data engineering field.experience in Big Data, Hadoop Ecosystem, Spark, Python, SQL, Hive,proficiency in relational SQL and NoSQL databases.Strong in writing & tuning the ETL Jobs, DB Programs and Queries, Python, Spark, Hive, ClouderaSound in Data Management Fundamentals and Data Architect, Modelling, Governance.Strong Domain Knowledge in Banking/Finance area like understanding of the various processes, products, and services within the banking industry, core functions, regulations, and operational aspects of banking institutions.Hands-on experience with Azure Cloud services such as Azure Data Factory (ADF), Databricks, and Cloud Storage.Experience in workflow management tools such as Airflow, Data streaming App such as Kafka, and CI/CD toolsManagement Skills:People management responsibilities encompass the tasks and activities involved in leading, guiding, and developing a team of individuals to achieve organizational goals effectively.Assigning tasks and responsibilities to team members based on their skills, strengths, and development needs.Follow predefined processes and workflows to carry out specific tasks and activities accurately and in a timely manner.Ensure the quality and accuracy of work by double-checking data entries and outputs, validating information, and resolving discrepancies.Analyze the needs, expectations, and concerns of each stakeholder. Assess their level of influence, interest, and potential power to affect the project's success.Additional Skills Desired:Good Communication SkillsStrong Analytical ThinkingAbility to handle stakeholders.Confident in decision making and the ability to explain processes or choices as needed.Ability to multitask on multiple projects in various phases.Project Methodology KnowledgeProgram Management & DeliverableComputer Science degree or equivalent