Data Engineer

Job PurposeThe Information Management stream delivers outcomes to help different parts of the bank gain meaningful insights from their Data Assets. Our core function spans multiple Hadoop, Business Intelligence and Analytics Cluster within Information Management. The work involves developing new requirements in Hadoop application across Enterprise Data Platform, analyzing the requirements, estimating the efforts, designing an effective solution, data architecture and management, change release management, resource management, handling the deliverables across the business groups in the Bank. Also, handling critical applications that influence bank’s management decision making process, regulatory reporting (which can influence financial/non-financial penalties) and operational reporting.3. Key Responsibilities:· Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity. · Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increase data accessibility, and fostering data-driven decision making across the organization. · Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes. · Writes unit/integration tests, contributes to engineering wiki, and documents work. · Designs and develops data integrations and data quality framework · Building ETL workflows, jobs or scripts or custom scripting languages to automate the data integration process. · Raising CR to the Data Platform team for production deployment. · Ensuring data integrity and quality within the Data Platform team. This involves setting up data validation checks, error handling mechanisms, and conducting regular audits to identify and correct data anomalies. · Optimizing the Data Platform's performance to meet SLAs and end-user requirements. This may involve fine-tuning SQL queries, optimizing ETL processes, or recommending hardware upgrades as needed. · Providing support and assistance to end-users, including data analysts, business intelligence teams, and other stakeholders. · Maintaining comprehensive documentation for the Data Platform environment, including data models, ETL processes, data dictionaries, and standard operating procedures. · Collaborate with Incident management team to track and report on incidents, their resolutions, and any necessary follow-up actions.5. Knowledge, Skills and ExperienceCore Skills: · At least 7+ years of experience in the Data engineering field. · 6+ years of experience in Big Data, Hadoop Ecosystem, Spark, Python, SQL, Hive, proficiency in relational SQL and NoSQL databases.· Strong in writing & tuning the ETL Jobs, DB Programs and Queries, Python, Spark, Hive, Cloudera · Sound in Data Management Fundamentals and Data Architect, Modelling, Governance. · Strong Domain Knowledge in Banking/Finance area like understanding of the various processes, products, and services within the banking industry, core functions, regulations, and operational aspects of banking institutions. · Hands-on experience with Azure Cloud services such as Azure Data Factory (ADF), Databricks, and Cloud Storage. · Experience in workflow management tools such as Airflow, Data streaming App such as Kafka, and CI/CD tools Management Skills: · People management responsibilities encompass the tasks and activities involved in leading, guiding, and developing a team of individuals to achieve organizational goals effectively. · Assigning tasks and responsibilities to team members based on their skills, strengths, and development needs. · Follow predefined processes and workflows to carry out specific tasks and activities accurately and in a timely manner. · Ensure the quality and accuracy of work by double-checking data entries and outputs, validating information, and resolving discrepancies. · Analyze the needs, expectations, and concerns of each stakeholder. Assess their level of influence, interest, and potential power to affect the project's success. Additional Skills Desired: · Good Communication Skills · Strong Analytical Thinking · Ability to handle stakeholders. · Confident in decision making and the ability to explain processes or choices as needed. · Ability to multitask on multiple projects in various phases. · Project Methodology Knowledge · Program Management & Deliverable · Computer Science degree or equivalent