Twitter github

Category “work”

Cloud Native Predictions for 2021 and Beyond

I hope everyone had a wonderful holiday break as the first couple weeks of January 2021 have been pretty wild, from insurrections to new COVID strains. In cloud native land, the CNCF recently released its annual report on all the work we accomplished last year. I recommend everyone take an opportunity to go through the report, we had a solid year given the wild pandemic circumstances.

https://twitter.com/CloudNativeFdn/status/1343914259177222145

As part of my job, I have a unique and privileged vantage point of cloud native trends given to all the member companies and developers I work with, so I figured I’d share my thoughts of where things will be going in 2021 and beyond:

Cloud Native IDEs

As a person who has spent a decent portion of his career working on developer tools inside the Eclipse Foundation, I am nothing but thrilled with the recent progress of the state of the art. The future will hold that the development lifecycle (code, build, debug) will happen mostly in the cloud versus your local Emacs or VSCode setup. You will end up getting a full dev environment setup for every pull request, pre-configured and connected to their own deployment to aid your development and debugging needs. A concrete example of this technology today is enabled via GitHub Codespaces and GitPod. While GitHub Codespaces is still in beta, you can try this experience live today with GitPod, using Prometheus as an example. In a minute or so, you have a completely live development environment with an editor and preview environment. The wild thing is that this development environment (workspace) is described in code and shareable with other developers on your team like any other code artifact.

In the end, I expect to see incredible innovation in the cloud native IDE space over the next year, especially as GitHub Codespaces enters out of beta and becomes more widely available so developers can experience this new concept and fall in love.

Kubernetes on the Edge

Kubernetes was born through usage across massive data centers but Kubernetes will evolve just like Linux did for new environments. What happened with Linux was that end users eventually stretched the kernel to support a variety of new deployment scenarios from mobile, embedded and more. I strongly believe Kubernetes will go through a similar evolution and we are already witnessing Telcos (and startups) explore Kubernetes as an edge platform through transforming VNFs into Cloud Native Network Functions (CNFs) along with open source projects like k3s, KubeEdge, k0s, LFEdge, Eclipse ioFog and more. The forces driving hyperscaler clouds to support telcos and the edge, combined with the ability to reuse cloud native software and build upon already a large ecosystem will cement Kubernetes as a dominant platform in edge computing over the next few years.

Cloud Native + Wasm

Web Assembly (Wasm) is a technology that is nascent but I expect it to become a growing utility and workload in the cloud native ecosystem especially as WASI matures and as Kubernetes is used more as an edge orchestrator as described previously. One use case is powering an extension mechanism, like what Envoy does with filters and LuaJIT. Instead of dealing with Lua directly, you can work with a smaller optimized runtime that supports a variety of programming languages. The Envoy project is currently in its journey in adopting Wasm and I expect a similar pattern to follow for any environment that scripting languages are a popular extension mechanism to be wholesale replaced by Wasm in the future.

On the Kubernetes front, there are projects like Krustlet from Microsoft that are exploring how a WASI-based runtime could be supported in Kubernetes. This shouldn’t be too surprising as Kubernetes is already being extended via CRDs and other mechanisms to run different types of workloads like VMs (KubeVirt) and more.

Also, if you’re new to Wasm, I recommend this new intro course from the Linux Foundation that goes over the space, along with the excellection documentation 

Rise of FinOps (CFM)

The coronavirus outbreak has accelerated the shift to cloud native. At least half of companies are accelerating their cloud plans amid the crisis… nearly 60% of respondents said cloud usage would exceed prior plans owing to the COVID-19 pandemic (State of the Cloud Report 2020). On top of that, Cloud Financial Management (or FinOps) is a growing issue and concern for many companies and honestly comes up in about half of my discussions the last six months with companies navigating their cloud native journey. You can also argue that cloud providers aren’t incentivized to make cloud financial management easier as that would make it easier for customers to spend less, however, the true pain is lack of open source innovation and standardization around cloud financial management in my opinion (all the clouds do cost management differently). In the CNCF context, there aren’t many open source projects trying to make FinOps easier, there is the KubeCost project but it’s fairly early days.

Also, the Linux Foundation recently launched the “FinOps Foundation” to help innovation in this space, they have some great introductory materials in this space. I expect to see a lot more open source projects and specifications in the FinOps space in the coming years.

More Rust in Cloud Native

Rust is still a young and niche programming language, especially if you look at programming language rankings from Redmonk as an example. However, my feeling is that you will see Rust in more cloud native projects over the coming year given that there are already a handful of CNCF projects taking advantage of Rust to it popping up in interesting infrastructure projects like the microvm Firecracker. While CNCF currently has a super majority of projects written in Golang, I expect Rust-based projects to be on par with Go-based ones in a couple of years as the Rust community matures.

GitOps + CD/PD Grows Significantly

GitOps is an operating model for cloud native technologies, providing a set of best practices that unify deployment, management and monitoring for applications (originally coined by Alexis Richardson from Weaveworks fame). The most important aspect of GitOps is describing the desired system state versioned in Git via a declaration fashion, that essentially enables a complex set of system changes to be applied correctly and then verified (via a nice audit log enabled via Git and other tools). From a pragmatic standpoint, GitOps improves developer experience and with the growth of projects like Argo, GitLab, Flux and so on, I expect GitOps tools to hit the enterprise more this year. If you look at the data from say GitLab, GitOps is still a nascent practice where the majority of companies haven’t explored it yet but as more companies move to adopt cloud native software at scale, GitOps will naturally follow in my opinion. If you’re interested in learning more about this space, I recommend checking out the newly formed GitOps Working Group in CNCF.

Service Catalogs 2.0: Cloud Native Developer Dashboards

The concept of a service catalog isn’t a new thing, for some of us older folks that grew up in the ITIL era you may remember things such as CMDBs (the horror). However, with the rise of microservices and cloud native development, the ability to catalog services and index a variety of real time service metadata is paramount to drive developer automation. This can include using a service catalog to understand ownership to handle incident management, manage SLOs and more. 

In the future, you will see a trend towards developer dashboards that are not only a service catalog, but provide an ability to extend the dashboard through a variety of automation features all in one place. The canonical open source examples of this are Backstage and Clutch from Lyft, however, any company with a fairly modern cloud native deployment tends to have a platform infrastructure team that has tried to build something similar. As the open source developer dashboards mature with a large plug-in ecosystem, you’ll see accelerated adoption by platform engineering teams everywhere.

Cross Cloud Becomes More Real

Kubernetes and the cloud native movement have demonstrated that cloud native and multi cloud approaches are possible in production environments, the data is clear that “93% of enterprises have a strategy to use multiple providers like Microsoft Azure, Amazon Web Services, and Google Cloud” (State of the Cloud Report 2020). The fact that Kubernetes has matured over the years along with the cloud market, will hopefully unlock programmatic cross-cloud managed services. A concrete example of this approach is embodied in the Crossplane project that provides an open source cross cloud control plane taking advantage of the Kubernetes API extensibility to enable cross cloud workload management (see “GitLab Deploys the Crossplane Control Plane to Offer Multicloud Deployments”).

Mainstream eBPF

eBPF allows you to run programs in the Linux Kernel without changing the kernel code or loading a module, you can think of it as a sandboxed extension mechanism. eBPF has allowed a new generation of software to extend the behavior of the Linux kernel to support a variety of different things from improved networking, monitoring and security. The downside of eBPF historically is that it requires a modern kernel version to take advantage of it and for a long time, that just wasn’t a realistic option for many companies. However, things are changing and even newer versions of RHEL finally support eBPF so you will see more projects take advantage. If you look at the latest container report from Sysdig, you can see the adoption of Falco rising recently which although the report may be a bit biased from Sysdig, it is reflected in production usage. So stay tuned and look for more eBPF based projects in the future!

Finally, Happy 2021!

I have a few more predictions and trends to share especially around end user driven open source, service mesh cannibalization/standardization, Prometheus+OTel, KYC for securing the software supply chain and more but I’ll save that for more detailed posts, nine predictions are enough to kick off the new year! Anyways, thanks for reading and I hope to see everyone at KubeCon + CloudNativeCon EU in May 2021, registration is open!

4 Years at The Linux Foundation

Late last year marked the 4th year anniversary of the formation of the CNCF and me joining The Linux Foundation:

As we enter 2020, it’s amusing for me to reflect on my decision to join The Linux Foundation a little over 4 years ago when I was looking for something new to focus on. I spent about 5 years at Twitter which felt like an eternity (the average tenure for a silicon valley employee is under 2 years), focused on open source and enjoyed the startup life of going from a hundred or so engineers to a couple of thousand. I truly enjoyed the ride, it was a high impact experience where we were able to open source projects that changed the industry for the better: Bootstrap (changed front end development for the better), Twemoji (made emojis more open source friendly and embeddable), Mesos (pushed the state of art for open source infrastructure), co-founded TODO Group (pushed the state of corporate open source programs forward) and more!

When I was looking for change, I wanted to find an opportunity that could impact more than I could just do at one company. I had some offers from FAANG companies and amazing startups but eventually settled on the nonprofit Linux Foundation because I wanted to build an open source foundation from scratch, teach other companies about open source best practices and assumed non profit life would be a bit more relaxing than diving into a new company (I was wrong). Also, I was throughly convinced that an openly governed foundation pushing Kubernetes, container specifications and adjacent independent cloud native technologies would be the right model to move open infrastructure forward.

As we enter 2020, I realize that I’ve been with one organization for a long time and that puts me on edge as I enjoy challenges, chaos and dread anything that makes me comfortable or complacent. Also, I have a strong desire to focus on efforts that involve improving the state of security and privacy in a connected world, participatory democracy, climate change; also anything that pushes open source to new industries and geographies.

While I’m always happy to entertain opportunities that align to my goals, the one thing that I do enjoy at the LF is that I’ve had the ability to build a variety of new open source foundations improving industries and communities: CDF, GraphQL Foundation, Open Container Initiative (OCI), Presto Foundation, TODO Group, Urban Computing Foundation and more.

Anyways, thanks for reading and I look forward to another year of bringing open source practices to new industries and places, the world is better when we are collaborating openly.

Open Source Gerrymandering

Over the years, I have spent a lot of time thinking about and working on open source communities… from bootstrapping projects out of corporations (or broken communities), to starting brand new open source foundations.

I was recently having a conversation with an old colleague about bringing an open source project out of a company into the wild and how to setup the project for success. A key part of that discussion involved setting up the governance for the project and what that means. There was also discussion how neutral and open governance under a nonprofit foundation can be good for certain projects as research has shown that neutral foundations can promote growth and community better than other approaches. Also the conversation led to a funny side discussion on the concept of gerrymandering and open source.

For those who aren’t familiar with the term, it’s become popular in the US political lexicon as a “practice intended to establish a political advantage for a particular party or group by manipulating district boundaries.” A practical example of this is from my town of Austin TX which is in district 35 which snakes all the way from Austin to San Antonio for some reason.

The same concept of gerrymandering can apply to open source communities as open source projects can act like mini political institutions (or bigger ones in the case of Kubernetes). I shared some of my favorite examples with my friend so I figured I’d write this down for future reference and share it with folks as you really need to read the “fine print” to find these at times.

Apache Cassandra

The Apache Software Foundation (ASF) is a fantastic open source organization that has been around for a long time (they celebrated their 20th anniversary) and has had a lot of impact across the world. The way projects are governed in the ASF are through the Apache Way, which places a lot of emphasis on “community over code” amongst some other principles which are great practices for open source projects to follow.

There have been some interesting governance issues and lessons learned over the years in the ASF, in particular it can be challenging when you have a strong single vendor associated with a project as was with the case with Cassandra awhile ago:

As the ASF board noted in the minutes from its meeting with DataStax representatives, “The Board expressed continuing concern that the PMC was not acting independently and that one company had undue influence over the project.” There was some interesting press around the time this happened:

“Jagielski told me in an interview, echoing what he’d said on the Cassandra mailing list, that undue influence conflicts with project leadership obligations established by the ASF. As he suggested, the ASF tried many times to get a DataStax-heavy Project Management Committee (PMC) to pay attention to alleged trademark and other violations, to no avail. Whatever DataStax’s positive influence on the development of the project—in other words—it failed to exercise equivalent influence on governing the project in ASF fashion.”

The ASF basically forced a reorganization of the Cassandra PMC to be in more in lines with its values and then caused the primary vendor behind the project to pull engineers off the open source project.

Containerd

The containerd project is an industry-standard container runtime with an emphasis on simplicity, robustness and portability. The history of the project comes from being born at Docker where their open source projects had a governance policy essentially aligned with the BDFL philosophy with one of their project founders.

In CNCF, (which containered is a project of), project governance documents aren’t considered static and evolve over time to meet the needs of their community. For example, when containerd joined the CNCF their governance was geared towards a BDFL approach but over time evolved to a more neutral approach that spread authority across maintainers.

Cloud Foundry

Cloud Foundry is an open source community that has a large and mature ecosystem of PaaS focused projects. In the Cloud Foundry Foundation (CFF), they have a unique governance clauses in regards to how affiliates are treated and voting.

Pivotal Platinum Director Voting Power. The Platinum Director appointed by Pivotal (“Pivotal Director”) shall have five (5) votes on any matter submitted to a vote of the Board. (i) On a date one (1) year after the incorporation date set forth in the Certificate, the number of Pivotal Director’s votes will be reduced to three (3). (ii) On a date two (2) years after the incorporation date set forth in the Certificate, the number of Pivotal Director’s votes will be reduced to one (1)

To bootstrap the foundation, the originating company wanted a little bit of control for a couple of years, which can make sense in some situations as the beginning of a foundation can be a tumultuous time. In my opinion, it’s great to see the extra vote clause expire after 2 years, however, it’s still very unfair to the early potential members of the organization.

Another example of open source gerrymandering can be how votes are represented by member companies that are owned by a single entity:

At no time may a Member and its Affiliates have more than one Director who is an employee, officer, director, or consultant of that Member, except that Pivotal, EMC, and VMware, though Affiliates, shall each have one (1) Director on the Board).

This is an interesting tidbit given that Dell owns Pivotal, EMC and VMWare. In some organizations, usually there is legal language that collapses owned entities into one vote.

I personally I’m not the biggest fan of this approach as it makes things unfair from the beginning and can be an impediment to wide adoption across the industry. There can definitely be reasons of why you need to do this in the formation phase but it should be done with caution. If you saw the recent news that Pivotal was being spun back into VMWare and their woes with adoption, it shouldn’t come as a surprise in my opinion as one company was bearing too much of the burden in my opinion and not building a diverse community of contributors.

Cloud Native Computing Foundation (CNCF)

If you remember the early days of the container and orchestration wars, there was a lot different technologies, approaches and corporate politics. When CNCF was founded, the original charter included a clause that upgraded certain startup members from Silver to Platinum that were important in the ever evolving cloud native ecosystem.

“The Governing Board may extend a Platinum membership at the Silver Membership Scale rates on a year-by-year basis for up to 5 years to startup companies with revenues less than $50 million that are deemed strategic technology contributors by the Governing Board.”

In my opinion, that particular piece in the charter was important in bringing together all the relevant startups to the table along with the big established companies at the time.

In terms of projects, the CNCF Technical Oversight Committee (TOC) defines a set of principles to steward the technical community. The most important principle is around a minimum viable governance that enables projects to be self-governing. TOC members are available to provide guidance to the projects but do not control them. 

https://twitter.com/CloudNativeFdn/status/1167455648768045056

Unlike Apache and the Apache Way, CNCF does not require its hosted projects to follow any specific governance model. Instead, CNCF specifies that graduated projects need to “explicitly define a project governance and committer process.” So in reality, CNCF operates under the principle of subsidiarity, encouraging decisions to be made at the lowest project level consistent with their resolution.

GitLab

GitLab is a fantastic open source project AND company that I admire deeply for their transparency. The way the GitLab project is structured is that it’s wholly owned by the GitLab company (they also own the trademark). To the credit of GitLab, they make this clear via their stewardship principles online and discuss what they consider enterprise product work versus project work.

I’d love for them in the future to separate the branding from the company, project and the product as I believe it’s confusing and dilutes the messaging, but that’s just my opinion 🙂

Istio

Istio is a popular service mesh project originated at Google. It has documented its governance model publicly: https://github.com/istio/community/blob/master/STEERING-COMMITTEE.md

However, as you can see, it’s heavily tilted towards Google and there seems to be no limits on the number of spots on the steering committee from one company which is a common tactic in open governance approaches to keep things fair. On top of that, Google owns the trademark, domains and other project assets so I’d consider Istio to be heavily gerrymandered in Google’s versus the community’s interest.

JCP

I had the pleasure of serving on the Java Community Process (JCP) Executive Committee for a few years while I was at Twitter. It’s a great organization that drives standardization across the Java ecosystem, some of the fine print is interesting though:

“The EC is composed of 25 Java Community Process Members whose seats are allocated as follows: 16 Ratified Seats, 6 Elected Seats, and 2 Associate Seats, plus one permanent seat held by Oracle. (Oracle’s representative must not be a member of the PMO.) The EC is led by a non-voting Chair from the PMO.”

This essentially gives Oracle a permanent seat on the Executive Committee.

Here’s another fun clause:

Ballots to approve Umbrella JSRs that define the initial version of a new Platform Edition Specification or JSRs that propose changes to the Java language are approved if (a) at least a two-thirds majority of the votes cast are “yes” votes, (b) a minimum of 5 “yes” votes are cast, and (c) Oracle casts one of the “yes” votes. Ballots are otherwise rejected.

This essentially gives Oracle a veto vote on any JSR.

Note: The coolest thing the JCP has done is contribute the EE specification work to the Eclipse Foundation and form the Jakarta project over there to steward things in an open way.

Knative

Knative, like Istio mentioned above, is an open source project that was born at Google and controlled by Google. There have been a lot of discussion lately about this as Google recently decided to not openly govern the project and move it to a neutral foundation:

Kubernetes

Kubernetes operates under the auspices of the CNCF and openly governed by the Kubernetes Steering Committee (KSC). The Kubernetes project has grown significantly over time, but has done a great job of keeping things openly governed and inclusive in my opinion, especially compared to its project size these days. The KSC governs the project along with a variety of sub working groups. Also, the Kubernetes trademark is neutrally owned by the CNCF and openly governed via the Conformance Working Group which decides how certification works for the community, which there are nearly 100 certified solutions out there!

Spinnaker

The Spinnaker project was originally born at Netflix and recently spun out into the Continuous Delivery Foundation (CDF) as an openly governed project. The project assets, from domains to github to trademarks are all neutrally owned by the community through the CDF.

Vault

Vault is a fantastic and widely used secrets management tool from Hashicorp. It’s a single vendor controlled open source project that has an open core model with an open source and enterprise versions (see matrix). What this essentially means is that the buck stops at the single vendor on what features/fixes end up in the open source version, most likely that won’t include things that they sell in their enterprise offering.

Conclusion

I hope you learned something new about open source projects, foundations and communities as these things can be a little bit more complicated as you dig into the details. It’s really important to note that there is a difference between open source and open governance and you should always be skeptical of a project that claims it’s truly open if only one for profit company owns all the assets and control. While there’s nothing wrong with this approach at all, most organizations don’t set expectations up front which can lead to frustrations down the road. Note, there’s nothing wrong with single vendor controlled open source projects, I think they are great but I think they need to be upfront, similar to what GitLab stewardship principles on what they will put in open source versus their enterprise version.

In conclusion, as with anything in life, you should always read the fine print of an open source communities charter or legal paperwork to understand how it works. The lesson here is that every organization or project has its own rules and governance and it’s important that you understand how decisions are made and who has ownership of project assets like trademarks.

KubeCon Shanghai 2019: Cloud Native at China Scale

Last month, we held KubeCon + Cloud NativeCon in Shanghai, one of the largest open source events in China. Recently, we published the conference transparency report detailing how the 3500 person event went but I’d like to share a couple take aways as someone who has been traveling to China quite a bit the last few years.

https://twitter.com/CloudNativeFdn/status/1154783210221527040

Cloud Native at China Scale

The scale of operating a popular service in China can be wild when you serve a billion users. This forces many of these companies to operate in a cloud native fashion, similar to their internet scale peers in Silicon Valley. I strongly believe we will see more interesting open source technology born in China over the next decade as they deal with scaling issues, similar to how a lot of internet scale open source technology was born from Google and other SV companies that had to support a larger user base. I highly recommend this great interview from Kevin Xu highlights some of the scale and open source projects coming out of China.

Also, Ant Financial joined CNCF as a Gold End User member, which is indicative of the trends of Chinese companies supporting open source they depend on and sharing the lessons they have learned. For those that aren’t aware, Ant Financial is the largest mobile payments company in the world (Alipay), running on Kubernetes and other CNCF projects. You can read their CNCF Case study here how they run on tens of thousands of Kubernetes nodes serving nearly a billion Alipay customers.

Giving Back: China: #2 contributor to Kubernetes

For those who aren’t aware, China is the second largest contributor to Kubernetes (and third to CNCF projects overall).

DevStats: https://k8s.devstats.cncf.io/d/50/countries-stats?orgId=1

It’s always great to have consumers contribute code back as that’s one important way how open source is sustainable in the long run.

On the whole, it was a fantastic to spend time with our cloud native community in Shanghai and we look forward to coming back to China next year, stay tuned for details for KubeCon + CloudNativeCon 2020 China! For now, we look forward to seeing many folks at KubeCon + CloudNativeCon 2019 San Diego which is gearing to be one of the largest open source events in the world.

Ways your company can support and sustain open source

Note: this article was original posted on https://opensource.com/article/19/4/ways-support-sustain-open-source

To make sure open source continues to thrive, we all need to find ways to sustain the communities and projects we depend on

The success of open source continues to grow; surveys show that the majority of companiesuse some form of open source, 99% of enterprises see open source as important, and almost half of developers are contributing back. It’s important to note that companies aren’t contributing to open source for purely altruistic reasons. Recent research from Harvard shows that open source-contributing companies capture up to 100% more productive value from open source than companies that do not contribute back. Another research study concluded countries adopting modern open source practices saw:

“a 0.6%–5.4% yearly increase in companies that use OSS, a 9%–18% yearly increase in the number of IT-related startups, a 6.6%–14% yearly increase in the number of individuals employed in IT related jobs, and a 5%–16% yearly decrease in software-related patents. All of these outcomes help to increase productivity and competitiveness at the national level. In aggregate, these results show that changes in government technology policy that favor OSS can have a positive impact on both global social value and domestic national competitiveness.”

In the end, there are many ways for a company or organization to sustain open source. It could be as simple as training your organization to contribute to open source projects you depend on or hiring engineers to work on open source projects. Here are eight ways your organization can contribute back to open source, based on examples in the industry.

Hire open source maintainers to work on open source

Companies with strategies to leverage open source often find the highest returns from hiring a maintainer of the projects they depend the most on. It’s no surprise if you look at the Who Writes the Linux Kernel report that the top contributors are all employed by companies like ARM, Google, Facebook, Intel, Red Hat, Samsung, and more.

Having a maintainer (full time or part time) on your staff can help your organization learn how to work within the project community and enable prioritization of upstream contributions based on understanding of what the community is focused on. Hiring the maintainers also means that the project will have people with enough time to focus on the details and the rigor that’s needed for a project to be useful; think security reviews, bug cleanup, release management, and more. A more predictable and reliable upstream project can benefit many in your organization while also improving the overall project community. As a bonus, maintainers can also become advocates for your organization and help with recruiting too!

Develop an open source award program or peer bonus fund

It is common for companies to have internal employee recognition programs that recognize individuals who go above and beyond. As an example, Red Hat has a community award program through Opensource.com. Some other companies have expanded their recognition programs to include open source contributors. For example, Google has an open source peer bonus program that recognizes external people who have made exceptional contributions to open source.

Start an open source program office

Many internet-scale companies, including Amazon, Google, Facebook, Twitter and more, have established formal open source programs (colloquially called OSPOs) within their organizations to manage open source strategy along with the consumption and contribution of open source.

If you want to increase your contributions to open source, research has shown that companies with formal open source programs are more likely to contribute back. If you want to learn from organizations with formal open source programs, I recommend you read the TODO Group Open Source Program Guides.

Launch an open source fund

Some organizations contribute fiscally to the open source projects that are important to them. For example, Comcast’s Open Source Development Grants “are intended to fund new or continued development of open source software in areas of interest to Comcast or of benefit to the Internet and broadband industries.” This isn’t just for big companies; small companies have open source funds, too. For example, CarGurus launched an open source fund and Eventbot is supporting open source with a small percentage of its company revenue. Another interesting approach is what Indeed has done by democratizing the open source funding process with its employees.

Contribute a portion of your company equity to open source

Consider donating a portion of your organization’s equity to an open source project you depend on. For example, Citus Data recently donated one percent of its equity to the PostgreSQL community. This worked out nicely; Citus Data was acquired by Microsoft recently, so the PostgreSQL community will benefit from that acquisition, too.

Support and join open source foundations

There are many open source foundations that house open source projects your organization depends on, including the Apache Foundation, Eclipse Foundation, Cloud Native Computing Foundation (home of Kubernetes), GraphQL Foundation, Let’s Encrypt, Linux Foundation, Open Source Initiative (OSI), OpenStack Foundation, NodeJS Foundation, and more.

https://twitter.com/CloudNativeFdn/status/1092544807900110848

Fund and participate in open source internships or retreats

There are many open source internship programs that you can participate in and help fund. Google Summer of Code (GSoC) is the largest, and it requires mentorship from employees who work on open source projects as part of the program. Or you can sponsor internships for underrepresented minorities in open source through Outreachy and CommunityBridge.

Another approach is to host an open source retreat at your company. For example, Stripe hosts open source retreats to contribute to open source projects it depends on.

Include open source in your corporate philanthropy initiatives

If your organization has a corporate sustainability or philanthropic arm, consider working with that team to include open source as a part of its work. For example, Bloomberg has a software philanthropy budget for projects it depends on, from Git to Eclipse to Python and more. In the future, I hope to see more corporate sustainability and philanthropy efforts—like Pledge 1%—that focus on funding critical open source infrastructure.

Conclusion

In conclusion, giving back to open source is not only the right thing to do, according to research, it’s also good for your business. To make sure open source continues to thrive and is sustainable in the long run, we all need to ensure that companies find ways to sustain the open source communities they depend on.

Troubles with the Open Source Gig Economy and Sustainability Tip Jar

There has been a lot of discussions lately on open source sustainability and honestly, it’s something I think about on a daily basis through my current role at The Linux Foundation and myself being an open source contributor and maintainer for ~20 years.

Open Source Tip Jars

There has been a trend lately to have open source communities/projects open an electronic tip jar for individuals (or organizations) to give back. While a tip jar can raise some funds, it’s truly not sustainable long term, as outside of it not being an effective way to raise funds for project development, it puts maintainers unfortunately in a gig style economy without health and retirement benefits. There are only a handful of individual maintainers are making money from donations, most tend to abandon the approach over time.

Take Patreon as an example, a popular funding platform for all sorts of things including open source projects that has been around since 2013 and has paid out $1 billion to creators since then. While that’s an astonishing figure, Patreon isn’t a sustainable business with its current private company model and level of VC funding of $105M:

“Under the company’s current business model, 90 percent of funds are paid directly to content creators. Patreon takes 5 percent, and the remaining 5 percent covers transaction fees.” Patreon CEO Jack Conte said in an interview with CNBC, that the platform will soon be facing the challenge of maintaining a profitable model as the company continues its growth.

In 2019, the company is also on track to pay out $500 million to content creators, 5% of that is $25 million and Patreon has ~300 employees, so probably not even covering their labor costs. Also, when the creators don’t have much say in the governance of the organization they depend on, especially if it’s a privately funded company, the terms can change overnight.

“So today, Patreon is overhauling its pricing. Any creator can still get a 5 percent rate, but just for a Lite version without bonus tools or different fan tiers. All of Patreon’s extra features will now be in the Pro plan, with an 8 percent rate, but with existing creators grandfathered in at 5 percent. And the new Premium enterprise plan for 12 percent (9 percent for existing creators) will offer full-service merchandise sales, multi-user team accounts and dedicated customer support. Patreon now has 3 million fans paying 100,000 creators more than half a billion dollars per year, and it will cross $1 billion in payouts in 2019 after six years in business. But Patreon was starving on its 5 percent rate, which some venture capitalists tell me is why they passed on its funding rounds totaling $105 million led by Thrive Capital and Index. Now it might make enough to keep the lights on, retain ownership and maybe even earn a profit one day.”

I guess we will see if Patreon is truly sustainable in the long run for its community and investors, it currently isn’t working out so well for most creators (“No one makes a living on Patreon”).

As a note, this also applies to smaller open source mini-patreon style fundraising efforts such as BountySource, OpenCollective, GitCoin, Fundition, Librepay and more. For example, I strongly admire OpenCollective and their transparency for a private VC funded company (on a side note, I also admire companies like Buffer who go further with transparency dashboards), from their level of funding as a startup to more. However, they are facing similar sustainability problems on a smaller scale if you look at the financials over time (their burn rate allows for 6-12 months depending on their growth), also maintainers don’t own any equity in the company. If this was a normal VC company the investors would cut their losses in my opinion.

It’s also interesting to see some nonprofit options out there like Liberapay founded in 2015 in France that is running on a break-even basis for all that I can tell. I think part of the sustainable solution for funding platforms is to build one in a way where it lives in a neutral place where creators have a say in governance and ownership. Outside of supporting projects forming their own companies, I’m interested in innovative startups like Tidelift which are experimenting with new approaches to make it easier for open source maintainers to get paid for their work by essentially by offering a subscription product and acting as a guarantor of sorts.

Note: I don’t think any of the blockchain coins will solve this problem or any problem for that matter.

Open Source Gig Economy

My biggest issue with these fundraising platforms is that even if we ever get to a point where some developers are making money, who pays for their health and retirement benefits (I know this is dependent on country but in many countries employers assist with retirement benefits). There has been a ton of press and research as of late on the gig economy, usually featuring Uber/Lyft in the news:

In the US and many countries, there are different protections for “employees” versus “independent contractors” and leads to a lot of situations where you lose protections depending on how you’re classified.

We can do better than having developers run in a hamster wheel of no benefits and protections offered via normal employment schemes. It’s not sustainable and not fair to developers.

Conclusion and Corporate Open Source Sustainability

I don’t think there is one solution for open source sustainability but I do strongly feel that VC-funded donation platforms that fuel the gig economy are not the best solutions for long term sustainability (even Patreon had to expand its business model outside just donations). There’s a part of me that feels some of this donation discussion echoes the past discussion of micropayments and publishers. This presentation from GitLab about how their business model evolved over time from the early days donations to consultancy to open core product business does a great job of showing the pitfalls of a variety of common approaches:

https://twitter.com/gitlab/status/1109213418840571905

Also, we should revisit the premise that open source isn’t sustainable to begin with, I think many folks in the community are selling short the progress that has been made over time in supporting open source. I’d like to make the argument that we are better off today than we were in the past. In fact, some research shows that “about 50% of all open source software development has been paid work for many years now and that many small projects are fully paid for by companies.” The reality is that there are still projects that fall through the cracks like OpenSSL that were aided eventually by the Core Infrastructure Initiative, so as long as we can discover and catch those we will be in good shape.

We should remember that a big part of innovation comes from developers working at organizations adopting open source software at scale and using it in interesting ways. It’s these organizations that should be tasked to sustain open source software versus individuals, especially since they depend on open source software to survive as a business. I also think it’s high time that corporate sustainability and social responsibility initiatives include open source contribution as part of them. I’ve seen some initiatives recently that I admire from organizations:

I truly hope to continue to see the trend of companies giving back to open source through contribution (fiscally or code) as I believe it’s key to open source sustainability and one of the reasons I co-founded the TODO Group and collaborate today with an excellent set of peers teaching organizations about the benefits of open source programs and why contributing back is good for business.

In the end, there are many ways to fund and sustain open source projects, from starting companies that sell open source based products and deliver value for customers, to working at organizations and having them dedicate time to contribute to open source and many more. I highly recommend visiting the oss.fund sheet to see what funding options are out there (also look at oss.cash to see large businesses built around open source technology). On a related note, I’m personally excited about the Community Bridge platform from the Linux Foundation which I was personally involved with:

CommunityBridge will automate some of the activities the Linux Foundation performs as an organization to benefit open source sustainability, from basic funding, expense reimbursement, mentorships and to paid internships that may lead to full time jobs with benefits.

Anyways, thanks for reading and I hope you learned something new about open source and understand why I think the charity/donation approach won’t work well for long term open source sustainability.

First KubeCon CloudNativeCon in China

After a week in Shanghai all I can say is wow, I’m truly humbled by the open source and cloud native community in China that showed up to support our first conference in China. I first want to thank the amazing CNCF events team and most importantly Janet Kuo and Liz Rice who acted as tireless program chairs for this first time event:

I’ve had the fortunate/unfortunate experience of traveling to China 6 times in the last 12 months and it’s been an experience learning about the open source community here. Also it’s been hilarious learning all the new tools like WeChat, Didi, Ofo and so on to navigate life in China but I can save that for another time (DiDi jail is the worst). The CNCF has grown from a few members in China to about 40 which represents a little more than 10% of the CNCF total membership. China is the third largest contributor to CNCF projects (in terms of contributors and committers) after the U.S. and Germany:

Huawei and PingCap lead the way for Chinese companies with 34,000+ and 32,000+ contributions respectively, and are the fifth and sixth largest contributors overall. We also now host three CNCF projects that were effectively born in China: Dragonfly (Alibaba), Harbor (VMWare China) and TiKV (PingCap). I’m proud of the work the CNCF has done to facilitate project learnings across the world as China scale open source is a trend that will continue to grow (I plan on writing more about this soon as I finalize my thoughts).

I’m also proud to award JD.com our first End User Award in China for their cultivation of cloud native in China, they run one of the largest bare metal Kubernetes and Vitess deployments in the world and have been very forthcoming in sharing the lessons around that experience:

Here are some of my other favorite tweets and moments from the conference:

https://twitter.com/KubernetesFin/status/1062886682670727168

https://twitter.com/zachorsarah/status/1062625760752525312

Anyway, thank you so much for everyone who attending and took a chance on our first event in China. I’m exhausted and heading for a long vacation but truly proud of the CNCF team and community for putting on an amazing event.

Red Hat and IBM: Elephants Can Dance

What an M&A surprise in the tech world yesterday with IBM picking up Red Hat, the jokes on Twitter were of course on point:

To expand on Alexis Richardson’s funny joke above, the clouds wars are no joke amongst the hyper scale clouds of the world and the war continues to escalate. Microsoft recently closed its deal with GitHub at $7.5B to only have IBM buy Red Hat for $34B. I can’t wait to see what Google, Oracle and the other large cloud providers pick up in the coming months.

I’ve had the privilege to work at both IBM and Red Hat earlier in my career so I’m familiar with the culture of both companies; it’s going to be interesting to see how the acquisition plays out over time. IBM is a gigantic company known for its bureaucracy that has been around for over 100 years and has successfully reinvented itself multiple times to survive (see the Who Says Elephants Can’t Dance book by IBM former CEO Lou Gerstner for a case study on this). Red Hat is an early open source pioneer with a fantastic and unique engineering culture that has been supporting remote work before it was cool and pioneered the concept of an “open source conflict of interest” clause (which will be interesting to see if IBM adopts):

“Participation in an open source community project, whether maintained by the Company or by another commercial or non-commercial entity or organization, does not constitute a conflict of interest even where you may make a determination in the interest of the project that is adverse to the Company’s interests”

There has been some FUD going around that IBM doesn’t fund open source or participate much in open source:

This FUD is absolutely crazy and needs to stop, IBM has arguably done more for open source than any other company to get where we are today with open source being prevalent in almost every industry and vertical:

IBM spent $1B on Linux before open source (and even Linux was cool)? Hell, I spent my early career working on open source at IBM where they had one of the first Open Source Program Offices (OSPOs) and spent my time hacking on Eclipse full time, which was another open source project that IBM helped start that disrupted the whole commercial tooling industrial complex. You can read more about IBM’s commitment to open source here which I think provides a great timeline of the various open source projects they have been involved in before open source was cool.

Anyways, to my Red Hat colleagues, my advice would be to give this a chance for awhile as IBM has a lot of strengths that Red Hat could take advantage of, they are truly a global company and have a solid sales channel that is embedded all over the world.

To my IBM colleagues, don’t “bluewash” this company and almost treat this as a reverse merger, embrace the culture from Red Hat and you should honestly consider making Jim Whitehurst CEO of IBM and Chris Wright CTO of IBM. As Lou Gerstner said, “culture isn’t just one aspect of the game, it is the game” and this is one area that Red Hat can greatly help IBM as it navigates towards the cloud.

Here are also a couple other good takes on the acquisition I enjoyed:

Finally, I’m really looking forward to see what IBM and Red Hat together, they have both been kindred spirits in making bets early on open source and I hope they bring that same zeal to the cloud. It at least makes my job running the Cloud Native Computing Foundation (CNCF) more entertaining 🙂

Developer Advocate Wars / Arms Race

While I’m not a huge fan of arms races, I have a unique perspective amongst open source foundations by having involvement from all the major cloud providers so I have an interesting vantage point in seeing a trend amongst them in ramping up developer advocacy hiring over the last couple of years. Furthermore, this hiring blitz is even more evident for me given the last couple of weeks have been conference crazy in the cloud native space with CloudNativeCon/KubeCon, MSBuild, Google IO and Red Hat Summit happening.

So what is happening with developer advocacy and the hyperscale cloud providers?

Note: If you aren’t familiar with the term developer advocate, I highly recommend this presentation by Patrick Chanzenon and blog from Ashley McNamara as an introduction to the profession.

Google

Google historically has been one of the early proponents of developer advocacy/relations and has invested in it since 2006 (see this excellent presentation for a historical perspective). A cursory search on LinkedIn shows a few hundred developer advocates employed at Google covering a variety of technologies from browsers to cloud. They also recently hired some heavy hitters like Sam Ramji and even Adam Seligman of Salesforce fame to help further build out their developer relations organization which was a smart move:

Anyways, I consider Google a leader in developer relations and with folks like Kelsey Hightower on staff, they are ahead of the curve.

I expect them to continue hiring like crazy to onboard more people onto their cloud offerings given their position in the market.

Microsoft

I used to work with Jeff Sandquist at Twitter and was delighted to hear he went back to Microsoft to build a developer advocacy focused organization. MS isn’t new at this game as some of us who are old enough and did MFC programming back in the day may remember Microsoft and their Channel9 advocacy program. Microsoft has been investing heavily in developer relations by making key hires (e.g., Ashley McNamara, Bridget Kromhout) and have rebooted their investment in open source to become one of the largest open source contributors in the industry. I’m not the first person to notice this trend by far, my friends at RedMonk noticed last year also. Searching the Microsoft job site, I see them hiring a handful of developer advocates mostly focused on cloud but they already have a large established roster of folks. It’s hard to get an exact count through the inexact science of flipping through LinkedIn search results, but I’d peg it 50-100 people currently employed doing developer relations.

Amazon (AWS)

In late 2016, Amazon started to expand their open source credibility by hiring folks like Adrian Cockroft, Zaheda Borat and Arun Gupta.

Using ancedata from my own experience being at events the last few years, Amazon has been aggressively speaking at conferences popular in the open source community. They have started to open source more examples and work with the community to improve them (e.g., https://github.com/aws-samples/aws-workshop-for-kubernetes)

It’s hard to get an exact number of people employed doing developer relations at AWS but going through the inexact science of LinkedIn search results, it looks to be in the 25-50 range which seems lower than I expected, but it’s hard as developer advocate is a fluid title (some people call themselves evangelists). In terms of hiring, searching the AWS jobs site I came across a couple of handful of developer advocate focused jobs so they seem to be aggressively hiring in this space. Also on a related note, I’m forever a fan of AWS for their execution and open discussion about hiring older engineers.

IBM

IBM traditionally has a strong position with their interactions with the open source community and even built a formal open source program within the company (I was also a fan of their DeveloperWorks initiative and even wrote technical articles for them in the past). They have also showed how making early bets on open source initiatives like Linux and Eclipse can reshape markets. In terms of developer advocacy, it seems about a year ago hired Jonas Jacobi to focus on developer relations from what I’ve been able to ascertain. They recently hosted an open source developer event in San Francisco called IBM Index which signals a desire to build developer communities outside the traditional IBM enterprise events like Think.

From a cursory LinkedIn search, IBM has 100+ developer advocates working on things from AI to Cloud. In terms of hiring, I’ve only found a handful of job descriptions currently open that are focused on developer advocacy. However, there were many job descriptions that were labeled software engineering that seemed to heavily focus on developer advocacy, for example, the folks that work on something called the IBM Cloud Garage seem to be a developer relations style job.

Oracle

Oracle has a burgeoning cloud business and according to some news sources, Oracle is actively trying to hire a new executive to run developer relations. A cursory job search shows that Oracle is ramping up their hiring, I see dozens of developer advocate positions open along with some more senior director level positions. I continue to expect Oracle to invest heavily in developer relations as this is an area they aren’t known for but has become table stakes in the public cloud business in my opinion.

Alibaba (and BAT+)

Alibaba and their cloud business is growing fast worldwide even though we don’t hear much about them in North America. In my opinion, they are making smart moves in Southeast Asia, India and other markets but that’s for another article to discuss. In my quest to learn about developer advocacy at Alibaba and even the other BAT companies, it was hard pressed to find any folks individually working on this. It’s clear to me that Alibaba has developer relations folks as if you go to their Alibaba Developers site there is some great content there but it’s been difficult to find current folks employed in devrel there and what they are hiring for. I’ll chalk this up to my inability to understand how job searching works in China and maybe one of my readers can shed some light on this. My hunch is that Alibaba will aggressively start hiring developer relations people worldwide if they didn’t start already, it will be fun to watch.

Conclusion

So what did I learn while I wrote this article? Well first off, getting hard numbers of how many job openings out there is a terrible science, sites like LinkedIn, Glassdoor and Indeed make it very difficult to extract data from their platforms to analyze. Second, developer relations is a nascent field and there’s no standard job description or career ladder. In many companies you have titles that range from “developer advocate” to “developer evangelist” to even “software engineer” that have primary responsibilities as developer relations. There have even been boutique communities popup to support people in the role of developer relations, see the Evangelist Collective slack as an example. Also the hiring of developer advocates isn’t just for large cloud providers or companies… you see some of the smart modern startups like Hashicorp, Turbine Labs and others hiring folks exclusively to make it easier to onboard developers to their tooling.

Finally, my prediction is that we will see cutthroat competition amongst the hyperscale cloud providers in hiring developer relations talent, especially out of the incumbents. The rest of the industry will learn from this experience that having a strong developer relations team is table stakes for any developer focused business. I’d love to write more on this topic but I’m short on time and I’m sure my friends at Redmonk can do a better job on this topic than me 🙂

Copenhagen: KubeCon and CloudNativeCon 2018 Takeaways

What a crazy week helping host our annual European community conference in Copenhagen… it’s been wild to see the community grow since the CNCF took over the stewardship of the conference (thank you Joseph Jacks, still remember those conversations we had in the early days):

I was also just blown away by the amount of CNCF ecosystem related jobs out there:

I have a few hours until I board my flight home so I figure I would share some of my take aways from the event in a series of tweets:

CNCF project adoption and the growth of the End User Community

The amount of end users I’ve bumped into at the conference was incredible, insurance companies, medical, automative, government, airlines, tax offices and more. In Dan Kohn’s keynote, he covered our official CNCF End User Community which was setup as a way to ensure End Users have a voice in CNCF governance:

CNCF has one of the largest, if not largest end user community membership of any open source foundation. I’m proud of what we built and mark my words, there will be a day when the number of official CNCF End Users will outnumber our vendors. Also, I was stoked to announce our first Top End User Award to Bloomberg showcasing one of our official end users using cloud native technology an interesting way:

If you’re using CNCF projects an interesting ways, I implore you to join our official End User Community so you have an official voice and more importantly, learn from other end users deploying CNCF projects.

May a thousand [kubernetes] operators bloom

In my opinion, one of the big themes of the conference was the rise of kubernetes operators. In Brandon Philips keynote, Red Hat (CoreOS) open sourced the Operator Framework which makes it easier to write and build operators:

At the conference itself, there were many projects and companies announcing operators for their project or product (see dotmesh, spark, NATS, vitess, etc), expect this trend to continue and explode over the next year, you can see the growing list of operators out there via awesome-operators repo.

CNCF is the center of open source serverless collaboration

The CNCF Serverless Working Group launched their first version of the CloudEvents project:

https://twitter.com/clemensv/status/992478395178119168

There was an incredible demo by Austen Collins showcasing the project across several cloud providers:

For an effort that started under a year ago, it’s nice to see Azure, Google, Oracle, IBM and other major cloud providers collaborate in the working group and support various open source serverless initiatives, I look forward to what they will do next:

CNCF 2020: Expanding ecosystem + Kubernetes: just run my code

Alexis Richardson gave a keynote outlining his thoughts on the future vision of CNCF which I found delightful for everyone who doesn’t attend every CNCF TOC meeting:

It’s not a surprise that I concur with a lot of these thoughts. In the bootstrapping days of CNCF, we were laying the foundation of projects required to bootstrap the ecosystem around Kubernetes and cloud native. The next step was increasing the reach of Kubernetes outside of just orchestration and focusing on technology areas as storage and security. The future of CNCF is all about increasing the mean time to developer satisfaction by improving the state of developer tooling. We need to get to the same point that developers are with Linux with Kubernetes, while super important foundational technology, developers don’t have to know the intimate details of how these systems work and instead stand on the shoulders of them to build their applications.

Another additional thing I’d like to mention that Alexis didn’t bring up formally.  One of my goals in CNCF is to ensure we build a sustainable ecosystem of projects, members and end users. As our ecosystem matures and some of our projects proverbially cross the chasm (we use the graduate parlance in CNCF)…

How do we ensure each of these parties are receiving value from their participation in the foundation? It’s something I think about on a daily basis as more CNCF projects get embedded everywhere, graduate and cross the chasm.

Kubernetes maturing and container standardization unlocks innovation

At the conference, Google open sourced gVisor as another approach to container runtimes which in my biased opinion is made possible due to OCI standardization efforts to allow this type of innovation without fear of breaking compatibility. As part of gVisor, runsc(like runc in OCI) provides an isolation boundary between the application and the host kernel and they have a ton of information in their README about the tradeoffs versus other container runtimes out there:

Conclusion

There are a lot more things to mention (e.g., rise of enovy and it becoming embedded everywhere, cloud native programming languages, chaos engineering) but I have to now board a flight home and get some sleep. Personally, I’m nothing but humbled by the growth of the community and conference the last few years, it’s been an honor helping build it out since the beginning. If you have any suggestions on improving the event or our community in anyway, please reach out via Twitter or shoot me an email:

We do listen to feedback and as an example, in Austin, people complained that the videos were taking too long to post and we aimed to have a quicker turn around this time and followed through with that in Copenhagen:

Thank you again for attending and see you in Shanghai and Seattle!