As a Lead Data Engineer, you will serve as a key architect in evolving our data infrastructure, under the technical direction of the Director of Data Engineering. Your primary responsibilities will include leading the design, implementation, and continuous enhancement of our data architectures, data pipelines, and interfaces to maximize scalability and efficiency. With advanced proficiency in SQL, Python, and cloud technologies (Azure, AWS, or Google Cloud), you will spearhead the development of... more details
Lead Data Engineer
Job LocationsUS-NY-New York
Job ID
2024-6283
Category
Information Technology
Minimum Rate
USD $150,000.00/Yr.
Maximum Rate
USD $160,000.00/Yr.
Work Location Type
Physical
Overview
At New York Blood Center Enterprises (NYBCe), one of the most comprehensive blood centers in the world, our focus is on cultivating excellence by merging cutting-edge innovation with diligent customer service, groundbreaking research, and comprehensive program and service development. Join us as we work towards meeting and exceeding the growing needs of our diverse communities, further our lifesaving strategic goals in a rapidly changing environment, and expand our impact on the local, national, and global communities we serve.
Responsibilities
As a Lead Data Engineer, you will serve as a key architect in evolving our data infrastructure, under the technical direction of the Director of Data Engineering. Your primary responsibilities will include leading the design, implementation, and continuous enhancement of our data architectures, data pipelines, and interfaces to maximize scalability and efficiency. With advanced proficiency in SQL, Python, and cloud technologies (Azure, AWS, or Google Cloud), you will spearhead the development of data solutions that are synchronized with our strategic objectives. Your deep understanding of Big Data concepts, including Spark and Cloud ETL tools like Databricks, will be crucial in addressing complex data challenges. Leveraging Agile/SCRUM methodologies, you will guide innovative and prompt project completions. This hands-on leadership role involves mentoring senior data engineers and fostering a culture of excellence and proactive growth. As a leading figure in our team, you will collaborate extensively with data scientists, BI teams, software engineers, and business stakeholders to devise and execute effective data engineering strategies that drive significant business impact.
Candidates must be able to report into one of the following NYBCe locations:New York City, NY; Kansas City, Missouri; St. Paul, Minnesota; Providence , RI and Newark, DE.
Primary Responsibilities:
Strategic Data Pipeline Design & Optimization: Lead the architectural design and optimization of robust, scalable data pipelines using SQL, Python, and cloud-based ETL tools such as Databricks. Oversee data flow and processing to support large-scale operations and ensure system-wide efficiency.
Advanced Data Modeling: Direct the development and refinement of complex data models to accurately mirror critical business processes, ensuring seamless integration with our comprehensive data architecture, including Big Data frameworks like Spark.
Data Architecture Leadership: Shape and enhance our data architecture strategy, making high-level decisions on data storage, consumption, integration, and management across cloud platforms (Azure, AWS, or Google Cloud).
Agile/SCRUM Leadership: Orchestrate Agile/SCRUM frameworks to drive efficient project delivery. Lead sprints and stand-ups, implementing these methodologies to optimize development processes across teams.
Senior-Level Collaboration: Lead collaboration efforts with data scientists, BI teams, and engineering groups to translate intricate data requirements into executable engineering strategies. Act as a primary technical liaison among stakeholders.
Mentorship of Senior Engineers: Mentor senior data engineers and technical team leads, instilling best practices in SQL, Python, and cloud technologies, while nurturing a culture of excellence and continuous advancement.
Quality & Governance Oversight: Establish and enforce the highest data quality standards and governance policies, ensuring reliability and regulatory compliance across all data operations.
Performance Tuning Expertise: Continuously monitor and refine the performance of data infrastructure, identifying and resolving high-level bottlenecks or inefficiencies in cloud and Big Data environments.
Innovation and Strategic Implementation: Stay at the forefront of emerging data engineering technologies and methodologies, leading the evaluation and strategic implementation of innovative tools and practices that enhance our capabilities.
Qualifications
Education:
Bachelor’s Degree in Computer Science, Data Science, Information Technology or other quantitative disciplines such as Science, Statistics, Economics, or Mathematics.
Essential Experience:
10+ years of progressive experience in data engineering, with proven expertise in designing, implementing, and leading initiatives for optimizing databases and data pipelines.
Extensive hands-on experience with SQL Server, Oracle, or other relational database management systems (RDBMS).
Advanced proficiency in SQL and Python for sophisticated data manipulation and analytics.
Demonstrated leadership in data modeling and architecture for both analytics and transactional systems within large-scale environments.
At least 2 years of experience leading teams of data engineers: Proven ability to mentor and guide teams in the development and optimization of data systems, fostering a collaborative and innovative working environment.
Cloud and Big Data Experience:
Expertise with at least one major cloud data platform (Azure, AWS, Google Cloud) with extensive application in data engineering projects.
Deep knowledge of Big Data technologies such as Spark and Cloud ETL tools like Databricks, with a focus on scalability and real-time processing capabilities.
Methodology and Tools:
Senior-level experience with Agile and SCRUM methodologies, leading successful project delivery in a dynamic development environment.
Skilled in developing data models for integration and analysis that support complex business intelligence and data analytics initiatives.
Any combination of education, training, and experience equivalent to the requirements above that has supplied the necessary knowledge, skills, and experience to perform the essential functions of the job.
Required Knowledge, Skills & Abilities:
Knowledge
Fluent Communication: Ability to articulate complex data concepts and project updates clearly to both technical and non-technical stakeholders.
Strong Data Analysis Ability: Expertise in analyzing large datasets to derive insights and inform business decisions.
Proficiency in SQL and Python: High level of skill in SQL and Python for data analysis, data manipulation and scripting.
ETL/ELT Architecture: In-depth knowledge of developing and managing ETL and ELT architectures using various tools and frameworks.
Cloud Experience: Experience with cloud platforms such as Azure, AWS, or Google Cloud, and their respective data services and tools.
Big Data Concepts: Understanding of Big Data technologies and frameworks, including Spark and Cloud ETL tools such as Databricks.
Agile and SCRUM Knowledge: Familiarity with Agile methodologies and SCRUM practices, capable of integrating these into project management and daily workflows.
Quality Assurance and Data Governance: Knowledge of data quality standards and governance, ensuring data integrity and compliance across all processes.
Skills
Collaboration: Ability to work effectively with cross-functional teams, including data scientists, BI analysts, and software engineers, to implement data solutions.
Mentorship and Leadership: Skills in mentoring junior engineers and leading project teams to promote knowledge sharing and professional growth within the team.
Innovation and Continuous Learning: Commitment to staying updated on the latest industry trends and technologies in data engineering and implementing them as relevant.
Abilities
Ability to interact with customers one-on-one or in large groups
Ability to work independently with remote supervision.
Ability to build in receiving feedback as part of the development process, and seek consistent and constructive feedback.
Ability to embrace accountability and ownership.
Peferred Qualifications:
Education: Master’s Degree in Computer Science, Data Science, Information Technology or other quantitative disciplines such as Science, Statistics, Economics, or Mathematics.
Experience with the Microsoft Azure technology stack or similar technologies in competing platforms.
Practical knowledge of data analytics and visualization tools to aid in data-driven decision making and reporting.
Certifications & Licenses:
Professional certification in Agile and SCRUM methodologies (e.g., Certified ScrumMaster (CSM), SAFe Agilist).
Certifications in Python and SQL programming (e.g., Microsoft Certified: Python Programming Specialist, Oracle SQL Certification).
Certifications in cloud services relevant to the job (e.g., AWS Certified Solutions Architect, Google Professional Data Engineer, Microsoft Certified: Azure Data Engineer Associate).
Big Data certifications (e.g., Cloudera Certified Professional (CCP): Data Engineer, Databricks Certified Professional Data Scientist).
Willing to attain certification, if not currently certified.
For applicants who will perform this position in New York City or Westchester County, the proposed annual salary is $150,000.00 p/yr. to $160,000.00 p/yr. For applicants who will perform this position outside of New York City or Westchester County, salary will reflect local market rates and be commensurate with the applicant’s skills, job-related knowledge, and experience.
Unless otherwise specified, all posted opportunities are located in the New York or Greater Tri-State office locations.
Job Abstracts is an independent Job Search Engine. Job Abstracts is not an agent or representative and is not endorsed, sponsored or affiliated with any employer. Job Abstracts uses proprietary technology to keep the availability and accuracy of its job listings and their details. All trademarks, service marks, logos, domain names, and job descriptions are the property of their respective holder. Job Abstracts does not have its members apply for a job on the jobabstracts.com website. Additionally, Job Abstracts may provide a list of third-party job listings that may not be affiliated with any employer. Please make sure you understand and agree to the website's Terms & Conditions and Privacy Policies you are applying on as they may differ from ours and are not in our control.
We would like to take a second to Welcome You to Job Abstracts, the nation’s largest Pure Job Board. With over 3.1 million job listings from 15,000+ Companies & Organizations, we help job searchers find careers that match their interests. As an anonymous user, you have probably discovered how easy our system is to use. However, you have just scratched the surface of what we can offer.
We encourage you to Register so you can use our most powerful features: searching with multiple terms, setting up multiple locations, establishing favorite companies, and accessing your search history. If you find a job you like, you can apply directly for it, and then, keep notes on it. We will also keep a lookout for jobs that match your search terms and email you when we find something you may like.
You can register for free and the system is free to use. If you like our system so far, click on Register and unlock the power required by serious job searchers.
Any time you conduct a search, the system shows you job matches, ranked by their Relevance Score (RS).
The score is calculated by a proprietary algorithm that uses Intelligent Machine Learning.
The Relevance Score tells you how well the job opportunity matches your search term or terms.
When not logged in, the system is limited to one search term. Scores for single term matches are usually lower.
When you register, log in, and set up multiple terms prioritized by importance, the jobs found for you will receive a much higher Relevance Score.
