Senior Software Engineer - Data Science Platform

Bloomberg runs on data. It's our business and our product. From the biggest banks to elite hedge funds, financial institutions need timely, accurate data to capture opportunities and evaluate risk in fast-moving markets. With petabytes of data available, a platform to transform and analyze the data is critical to our success.

Bloomberg’s Data Science Platform was established to support development efforts around data-driven science, machine learning, and business analytics. The platform aims to provide scalable compute, specialized hardware and first-class support for a variety of workloads such as Spark, Tensorflow and Jupyter. The platform was developed to provide a standard set of tooling for addressing the Model Development Life Cycle from experimentation and training to inference. It provides advanced features such as Hyperparameter Tuning as a Service and is beginning to invest in Model Management and Governance. The platform is built leveraging containerization, container orchestration and cloud architecture and built on top of 100% open source foundations.
   
The platform is poised for enormous user growth this year and has an ambitious roadmap in terms of new features as well as improved user experience. That’s where you come in. As a member of the multi-disciplinary Data Science Platform team, you’ll have the opportunity to make key technical decisions to keep this platform moving forward. 
  
Our team makes extensive use of open source (e.g. Kubernetes, Tensorflow, Spark and Jupyter) and is deeply involved in a number of communities. As part of that, we regularly upstream features we develop, present at conferences and collaborate with our peers in the industry. We are contributors to the Kubeflow project as well as founding members of the KFServing subproject to standardize ML Inference within the Kubernetes ecosystem. For Spark, we have implemented a scalable and resilient external shuffle service for dynamic resource allocation, a pluggable interface for secure worker creation, and a token renewal service that handles privacy and security across jobs, all in line with our effort to improve security and elasticity for Spark on Kubernetes. Open source is at the heart of our team. It's not just something we do in our free time, it is how we work.
 
We’ll trust you to:

• Interact with data scientists to understand their workflows and requirements to inform the next set of features for the platform
• Design solutions for problems such as elastic load distribution, GPU sharing and guaranteed scheduling
• Automate operation and improve telemetry of data science platform components in our infrastructure stack
 
You’ll need to be able to:

• Troubleshoot and debug run-time issues
• Provide developer and operational documentation
• Provide performance analysis and capacity planning for clusters
• Be organized and multi-task in a fast paced environment
• Have a passion for providing reliable and scalable infrastructure
 
You’ll need to have: 

• Experience with distributed systems eg. Kubernetes, Kafka, Zookeeper, Spark
• Proficiency in two or more languages (Python, Go, C++, Java, Scala, or JavaScript) and willingness to learn more as needed
• Linux systems experience (Network, OS, Filesystems)

• Experience building and scaling Docker-based systems using Kubernetes, Swarm, Rancher, Mesos
• Experience working with authentication & authorization systems such as Kerberos and LDAP
• Experience working with GPU compute software and hardware
• Ability to identify and perform OS and hardware-level optimizations
• Open source involvement such as a well-curated blog, accepted contribution, or community presence
• Experience with cloud providers such as AWS, GCP or Azure
• Experience with configuration management systems (Chef, Puppet, Ansible, or Salt)
• Experience with continuous integration tools and technologies (Jenkins, Git, Chat-ops)

 
If this sounds like you, apply! You can also learn more about our work using the links below:
 


• Machine Learning the Kubernetes Way - https://www.youtube.com/watch?v=ncED2EMcxZ8
• Inference with KFServing - https://www.youtube.com/watch?v=saMkA4fIOH8
• ML at Bloomberg - https://on-demand-gtc.gputechconf.com/gtcnew/sessionview.php?sessionName=s9810-machine+learning+%40+bloomberg%3a+building+on+kubernetes 
• Scaling Spark on Kubernetes -https://www.youtube.com/watch?v=GbpMOaSlMJ4

Bloomberg is an equal opportunities employer, and we value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.