Senior Software Engineer - Spark Runtimes
Bloomberg runs on data. It's our business and our product. From the biggest banks to the most elite hedge funds, financial institutions need timely, accurate data to capture opportunities and evaluate risk in fast-moving markets. With petabytes of data available, a platform to transform and analyze data is critical to our success.
Bloomberg has always focused on providing tenants with secure, reliable, and scalable solutions for their ETL pipelines. The Spark Runtimes team aims to provide scalable compute, specialized hardware, and first-class support for a variety of Apache Spark workloads and runtimes on top of Kubernetes. This platform is built leveraging containerization, container orchestration, and cloud architecture and built on top of 100% open source foundations.
The Spark Runtimes team additionally functions as a single-window clearinghouse for evaluating use-cases, determining fit, and assisting teams with using Spark, the way it's meant to be used. In joining this team of Spark experts, you will help define best practices around infrastructure management, software deployments, and security as we provide support for a powerful, flexible, and centralized compute infrastructure; address feature gaps across similar use cases through Spark core contributions, and commit to Kubernetes as the future for compute. On a daily basis, team members consult on projects in a wide variety of application domains and help provide design and implementation guidance as well as build self-service tooling to help with the instrumentation, profiling, and optimization of distributed financial applications.
The platform is poised for enormous user growth this year and has an ambitious roadmap in terms of new features as well as improved user experience. That’s where you come in. As a member of the multi-disciplinary Spark Runtimes team, you’ll have the opportunity to make key technical decisions to keep this platform moving forward.
Our team makes extensive use of open-source (e.g. Spark, Kubernetes, Istio) technologies and is deeply involved in a number of communities. As part of that, we regularly upstream features we develop, present at conferences, and collaborate with our peers in the industry. For Spark, we recently have implemented a scalable and resilient external shuffle service for dynamic resource allocation, a pluggable interface for secure worker creation, in a Kubernetes environment, and a token renewal service that handles privacy and security across Spark jobs, all in line with our effort to improve security and elasticity for Spark on Kubernetes. We have also contributed code for topology-aware RDD block replication as well as Spark ML bug fixes. Open source is at the heart of our team. It's not just something we do in our free time, it is how we work.
We'll trust you to:
You’ll need to be able to:
You'll need to have:
We'd love to see:
If this sounds like you, submit an application and watch our Spark Summit / KubeCon talks to learn more about some of the ideas that underpin our infrastructure offering:
Bloomberg is an equal opportunities employer, and we value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.
Mehr Jobangebote von Bloomberg L.P.
Weitere Angebote im Stellenmarkt