Senior Software Engineer - PACE Runtimes

Senior Software Engineer - PACE Runtimes The PACE (Parallel Analytics Compute Engine) Runtimes team is building low-latency, distributed analytics infrastructure for Bloomberg based on Apache Spark. We support critical and highly visible applications doing distributed financial calculations over large data sets. The team is looking for senior developers with interest and expertise in Spark. The team's objective is to foster Spark adoption at Bloomberg by providing a self-serve environment with good resilience and auto-scale properties, allowing application developers to focus on solving business problems. Our platform provides standardized building blocks that handle aspects such as data lifecycle management through connectivity to a variety of data sources and service endpoints. The platform also provides tools to create reusable libraries for distributed financial calculations for applications such as currency conversion or backtesting functions. We are currently focused on Kubernetes orchestration of Spark, re-architecting our platform with cloud architecture in mind. On a daily basis, team members consult on projects in a wide-variety of application domains and help provide design and implementation guidance as well as build self-service tooling to help with the instrumentation, profiling and optimization of distributed financial applications. We are also making contributions to Spark: recently we have implemented a scalable and resilient external shuffle service for dynamic resource allocation, a pluggable interface for secure worker creation, and a token renewal service that handles privacy and security across Spark jobs, all in line with our effort to improve security and elasticity for Spark on Kubernetes. We have also contributed code for topology-aware RDD block replication as well as Spark ML bug fixes. If you are a software engineer who is passionate about distributed systems, has seen how they behave in production and has expertise in developing backend software infrastructure, we'd like to talk to you about an opening on our team. We'll trust you to: - Interact with application teams who use our platform (or are looking to use it) to understand their requirements. Translate these requirements into common features that our platform should offer, removing cognitive burden for our users, and allowing them to focus on quickly solving business problems. - Contribute to the development of the PACE Runtimes core infrastructure, its integration with a variety of key Bloomberg data sources and services, as well as tooling to improve the platform user experience. - Guide engineers across Bloomberg with leveraging Spark and distributed data processing idioms to bring scale and capacity to some of the most complex and demanding financial applications in the organization. You'll need to have: - Experience designing and implementing low-latency, high-scalability systems. - 3+ years of experience programming in Java, Scala and Python. - A good understanding of Apache Spark programming models and constructs. - A solid understanding of distributed systems fundamentals. We'd love to see: - Proficiency in Scala and functional programming idioms. - An expert-level understanding of Apache Spark (your work may involve modifying its internals). - Experience with Kubernetes and its broader ecosystem (custom operators, service meshes etc.) - Professional experience developing software infrastructure and application frameworks. If this sounds like you, submit an application and watch our Spark Summit talks to learn more about some of the ideas that underpin our analytics runtime: - https://spark-summit.org/east-2016/events/spark-at-bloomberg/ - https://www.techatbloomberg.com/events/spark-summit-east-2017/ - https://databricks.com/session/apache-spark-on-k8s-and-hdfs-security - https://kccnceu19.sched.com/event/MPal/scaling-and-securing-spark-on-kubernetes-at-bloomberg-ilan-filonenko-bloomberg Bloomberg is an equal opportunities employer, and we value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.