Production engineering is a field that involves crafting, building, and maintaining large-scale production systems with high efficiency and availability. It encompasses various areas, including software and systems engineering practices, storage, data management, and services. Professionals in the role of Production Engineers hold specialized knowledge and expertise across various domains, including storage architecture, high-performance distributed storage, data management, systems, networking, coding, database management, prioritization, continuous delivery and deployment, along with open-source cloud-enabling technologies such as Kubernetes, containers, and virtualization. Their responsibilities include ensuring storage architectures are reliable, scalable, and efficient. They optimize data placement and access patterns. They manage large-scale distributed storage systems and ensure low-latency data access for HPC and AI/ML workloads.
Storage Production Engineers at NVIDIA ensure that our internal and external-facing GPU cloud services meet reliability and uptime goals as promised to the users while enabling developers to make changes to the existing system through careful preparation and planning while keeping an eye on capacity, latency, and performance. This role also requires a mindset focused on automating storage operations, improving data access efficiency, and optimizing storage performance. Much of our software development focuses on optimizing operations through automation, enhancing system responsiveness, and improving the efficiency of storage and production systems. Since Production Engineers are responsible for the big picture of how our systems interface with each other, we use a breadth of tools and approaches to tackle a broad spectrum of challenges. Practices such as proactive storage performance monitoring, automated fault detection and remediation, scalable data redundancy methods, and integration of intelligent caching mechanisms factor into iterative improvements that are key to system reliability and efficiency.
What You Will Be Doing
- Design, implement, and support large-scale storage clusters, ensuring scalability, high availability, and data integrity.
- Develop and maintain storage monitoring, logging, and alerting systems to ensure proactive detection and resolution of performance issues.
- Work with AI/ML workloads to improve storage architectures for low-latency access, efficient caching, and high-throughput performance.
- Improve the lifecycle of storage services – from inception and design to deployment, operation, and continuous optimization. Support storage services before they become available through activities such as system build consulting, developing automation frameworks, capacity management, and launch reviews.
- Maintain production storage infrastructure by supervising availability, latency, and system health, leveraging predictive analytics and AI-driven automation.
- Optimize storage efficiency through compression, deduplication, tiering strategies, and intelligent workload placement.
- Scale storage systems sustainably using AI/ML-driven automation, policy-based tiering, and dynamic data migration techniques. Ensure data security and compliance by implementing encryption, access controls, and auditing mechanisms for storage systems.
- Practice sustainable incident response and blameless root cause analysis. Be part of an on-call rotation to support storage and production systems.
What We Need To See
- BS degree or equivalent experience in Computer Science, Storage Systems, or a related technical field with 8+ years of practical experience.
- Experience with distributed and high-performance storage solutions, including clustered and parallel file systems, distributed object storage, and enterprise-grade storage systems.
- Solid understanding of block, file, and object storage technologies, including their scalability, reliability, and performance characteristics and standard processes.
- Experience with storage networking protocols such as NFS, SMB, iSCSI, S3, Fibre Channel, RDMA, and NVMe over Fabrics.
- Expertise in algorithms, data structures, complexity analysis, software design, and automating maintenance of large-scale Linux-based storage systems.
- Experience in one or more of the following: C/C++, Java, Python, Go, NodeJS, and Bash for storage automation, monitoring, and performance tuning.
- Hands-on experience with infrastructure configuration management tools like Ansible, Chef, Puppet, and Terraform for automating storage deployments. Experience with observability and tracing tools like InfluxDB, Prometheus, Grafana, and the Elastic stack for monitoring storage system health.
- You should possess excellent written and oral communication skills, excellent work ethics, a deep sense of teamwork, love to produce quality work and commitment to finishing your tasks every single day.
Ways to stand out from the crowd
- Deep understanding of extensive distributed storage systems, replication strategies, and erasure coding techniques. Experience in capacity planning, performance tuning, and troubleshooting high-throughput storage systems.
- Experience with Git, code review, pipelines, and CI/CD for handling infrastructure as code. Experience in analyzing and improving distributed storage system performance at scale. Strong debugging skills with a systematic problem-solving approach to identify sophisticated storage issues. Proven understanding of network protocols, architectures, and troubleshooting techniques, especially as it relates to storage performance, stability, and availability.
- Experience using or operating private and public cloud storage solutions based on Kubernetes, OpenStack, or hybrid cloud architectures. Ability to design and implement automated storage migration, backup, and disaster recovery strategies. Thrive in collaborative environments and enjoy working with various teams to optimize storage performance. Flexible in adapting to different working styles and emerging storage technologies.