Open Source at Bloomberg: 1H 2023 Engagement

Sep 28, 2023 10:00 AM ET

" " — During Ceph Days NYC, which was hosted at Bloomberg’s Park Avenue office in February 2023, Senior Software Engineer Nathan Hoad of Bloomberg’s Storage Engineering team talking about why the team built a “Message-Driven Telemetry System At Scale” for our Ceph clusters

Originally published on bloomberg.com

Many Bloomberg engineers are active consumers, community participants, contributors, and leaders of various open source projects. Over the first half of 2023, Bloomberg’s Open Source Program Office (OSPO) and its engineers have come together to build, maintain, develop, and fund open source projects. These activities range from filing new commits, contributing pull requests, developing and publishing tools and features, being appointed to and/or sponsoring leadership roles, launching the Bloomberg FOSS (Free and Open Source Software) Contributor Fund, and more. The Bloomberg team also regularly promotes its use cases for these projects and its efforts to advance open source technologies through numerous conference presentations, podcast interviews, and media engagements.

Our long track record of embracing open source is an investment in the future of these technologies. Our goal: to keep the open source ecosystem secure, resilient, and healthy as more and more users and companies rely on open source to keep their systems running.

Here is a roundup of Bloomberg’s open source engagement during the first half of 2023.

Contributions and Projects

Ceph: Bloomberg’s Distributed Storage team had its first major commit accepted in upstream Ceph; their PR fixed a major bug with race condition in multisite during full sync involving deletes.
Hera: A new version (v5) of Dyno’s open source project Hera was released in collaboration with Bloomberg and AI Software Engineer Sambhav Kothari. This release expanded Hera to feature parity with workflows and events.
Lexical: Several Bloomberg engineers have been contributing to this popular JavaScript Text Editor Framework that powers the Bloomberg Terminal. Between them, Attila Novak, Ben Carleton, Ebad Salehi, Ivaylo Pavlov, and Shubhanker Srivastava have landed 20 PRs in the first half of 2023 to improve core functionality.
Memray: Memray 1.8.0 was published, and it includes a groundbreaking feature: temporal flame graphs. These provide a new perspective on memory usage in programs with dynamic time range analysis. No more static snapshots; explore memory allocation over time!
MiniLMv2: Bloomberg’s AI researchers published “MiniLMv2.BB,” an open source implementation of the MiniLMv2 method detailed in the ACL 2021 paper “MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers.” Using this code, one can distill NLP language models, including BERT or RoBERTa, to a smaller size in order to enable lower latency inference.
Network Transport Framework (NTF): Bloomberg’s engineers published an open source version of its NTF on GitHub. This collection of asynchronous networking APIs/libraries is designed for sending and receiving data between processes in scalable, high-performance applications.
PyStack: Bloomberg’s Python Infrastructure team published PyStack, an open source debugging tool that can inspect the stack frames of a running Python process or core file to determine what is happening without having to understand CPython internals. [read more about PyStack]
TypeScript: Bloomberg’s JavaScript Infrastructure team continues to collaborate with the core TypeScript team to improve the language and tooling. Chi Leung helped modernize namespaces and Leo Elmecker-Plakolm introduced types for an ES2023 feature. There is also exciting progress on experiments to speed up type-checking via parallel builds.
Ecma TC39: Bloomberg contributes to governance, maintenance, and technical advancement of the JavaScript language. Daniel Ehrenberg is Ecma Vice President and Rob Palmer is Co-Chair of TC39. Software Engineer Ashley Claymore introduced the Await Dictionary proposal. Peter Klecha advanced the Promise.withResolvers() proposal to Stage 3. The Symbols as WeakMap keys and Change Array by Copy features are part of ES2023 thanks to Bloomberg engineers.
V8: Bloomberg’s ongoing collaboration with Igalia has led to significant wins for the developer experience via contributions to Google’s V8 JavaScript engine. Memory debugging via Heap Snapshots was optimized to be 2x-100x faster. Advanced native profiling on Windows using ETW tracing was enabled to show both native and JavaScript function names.

Community Engagement

Joined the Adoptium Working Group as an Enterprise Member (and also became a Contributing Member of the Eclipse Foundation) [read more].
Hosted Ceph Days NYC at its 120 Park Avenue office, where Nathan Hoad of our Storage Engineering team talked about why Bloomberg built a “Message-Driven Telemetry System At Scale” for our Ceph clusters. Our Head of Storage Engineering Matthew Leonard offered the opening remarks. [watch videos of all the Ceph Day NYC talks]
Sponsored the NYC++ Meetup, which provides an opportunity for the area’s C++ enthusiasts to socialize, network, and learn together.
Launched a new FOSS Contributor Fund, through a partnership between Bloomberg’s OSPO and Corporate Philanthropy teams, to support the people and communities who maintain open source projects. The fund will run a quarterly election to award up to three grants of $10,000 during each voting cycle. In Q1 2023, the first three projects to receive grants from the Fund were Apache Arrow, Celery, and Curl. The Q2 2023 grant recipients are MkDocs, OpenSSH, and PostgreSQL.
Sponsored the Python Software Foundation’s hiring of a full-time Deputy Developer-in-Residence to accelerate the velocity of the CPython core dev team. [learn more about the role]
Pradyun Gedam, an engineer with our Python Infrastructure team, was promoted to CPython core dev. He’s been active in the Python packaging ecosystem since 2017, and works on pip, PyPI, PyPA, Sphinx, TOML, and more. [learn more about Pradyun on Mouse Vs. Python]

Conference Sightings

Watch some of the many talks Bloomberg’s engineers delivered at conferences around the globe in the first half of 2023.

ACCU Conference 2023

Removing Needless Undefined Behavior for a Safer C++ | Alisdair Meredith
Managing External API’s in Enterprise Systems | Peter Muldoon
Rust for the Recalcitrant C++ Programmer | CB Bailey
Improving Compilation Times: Tools & Techniques | Vittorio Romeo
C++ Trivial Relocation Through Time | Mungo Gill
How Bloomberg uses Behavior-Driven Development (BDD) to Test Distributed Systems | Jacqueline Pan
Monads in Modern C++ | Georgi Koyrushki & Alistair Fisher
Introduction to Epoch-Based Memory Reclamation | Jeffrey Mendelsohn
Function Contracts in Practice using C++ | Rostislav Khlebnikov
What I Learned from Sockets: Applying the Unix Readiness Model When Composing Concurrent Operations in C++ | Filipp Gelman
Implementing a C++ Coroutine Task from Scratch | Dietmar Kühl

Berlin Buzzwords 2023

No Mean Feat: Upgrading a Customized Solr to Upstream Solr | Shikhar Srivastava

Cephalocon 2023

Ceph Multi-Site at Scale: Bloomberg’s Disaster Recovery Journey | Krunal Chheda and Jane Zhu
Over a Billion Requests Served per Day: Ensuring Everyone Is Happy with Our Ceph Clusters’ Performance | Nathan Hoad and Alex Wojno
Day 2 Keynote: The Road to Open Source Ceph: Bloomberg’s Enterprise Ceph Journey | Matthew Leonard
Embracing Ceph: Key Factors Driving Companies to Adopt Ceph Storage Solutions (Panel) | Matthew Leonard

KubeCon + CloudNativeCon Europe 2023

Customizing Your Buildpacks Build – Yes You Can! | Bloomberg’s Aidan Delaney and VMWare’s Natalie Arellano
The State and Future of Cloud-Native Model Serving | Bloomberg’s Dan Sun and Amazon’s Theofilos Papapanagiotou

Open Source Summit North America 2023

What Our OSPOs Have Learned About Measuring Project Health | Alyssa Wright

OpenInfra Summit Vancouver 2023

The Tortoise Beats the Hare: Upgrading the Operating System of the Cloud at Scale | Jin Geng and Tyler Stachecki
The OpenInfra Foundation named Bloomberg winner of the 2023 Superuser Awards, recognizing its adoption of and contributions to Linux, OpenStack, Ceph, and Kubernetes infrastructure. The Superuser Awards recognize organizations who use open infrastructure to improve their business while also contributing back to the open infrastructure community.

PyCon US 2023