Twitter github

Category “work”

@ApacheParquet Graduating and Mesos with Siri

The last week for me has been fun in open source land outside of me getting two of my wisdom teeth pulled out of my face. On the bright side, I have some pain killers now and also, two notable things happened. First it was nice to finally graduate Parquet out of the Apache Incubator:

It’s been a little over two years since we (Twitter) announced the open source columnar storage project with Cloudera. It’s a great feeling to see a plan come together and see this project grow over the years with 60+ contributors while hitting the notable achievement of graduating out of the Apache incubator gauntlet. If there’s any lesson here for me, it’s much easier to build an open source community when you do it an independent fashion with at least someone else in the beginning (thanks Cloudera).

Another notable thing that happened was that Apple finally announced that they are using Mesos to power Siri’s massive infrastructure.

In my experience of building open source communities, there are usually your public adopters and private adopters. There are companies that wish to remain private about the software they use at times and that’s fine, it’s understandable when it can be viewed as a competitive advantage. The challenge is how you work with these private adopters when they use an open source project of yours while wanting to collaborate behind the scenes.

Anyways, it’s a great feeling to see Apple opening up a bit about their infrastructure and open source usage after working with them for awhile. Hopefully this is a sign of things to come from them. Also, it would be nice if Apple just updated Siri so when you ask what Mesos is, it replies with a funny response and proclaims her love of open source infrastructure technology.

Overall, it’s been a great last week.

Thoughts on running an open source program (via @TODOGroup)

I recently posted on the @TODOGroup blog on why we run an open source program at Twitter:

Outside of just my experience, it’s been great to see other companies participating in this effort so you can hear from them too on running open source programs:

There should be more blog posts from us in the future about this topic, on top of us discussing other issues pertinent to companies working with open source communities.

Anyways, I hope you learned something new from these posts and if you’re a fan of company open source programs, please consider pushing your company or others to establish an official open source program or office. We all should give back as it’s in our best interest.

FoundationDB and Open Source Foundations

Just like any other day, I saw a funny tweet across my timeline this morning:

Not sure if you heard the news, but FoundationDB was a company with a decent amount of open source projects around a NoSQL database. They recently were bought by a much larger company and decided to close down the project, including removing the source and binaries from distribution channels.

I don’t want to sound like a broken record, but open source foundations are really useful. As a consumer, it helps you ensure that there is an independent governance structure in place along with fair ownership of the code (usually the foundation). This means that code won’t disappear overnight since an independent entity owns it, along with the broader community. From a producer point of view, you can build diversity in ownership and committers which will help you in the long run in building a sustainable open source community.

Oh well, c’est la vie.

Quick update and nice set of genuine tweets from the CouchDB folks who saw the benefits of having the code exist at an open source foundation:

Linux Kernel’s Code of Conflict

The Linux Kernel recently has come up with an aptly named “Code of Conflict” to deal with some of the criticism inside that community:

There’s also an interview from the Linux Foundation Executive Director, Jim Zemlin about this topic:

While not perfect in my opinion, it is the step in the right direction to ensure a well behaving community. While most people will be excellent to each other, there will also be outliers. Also, it’s important to set expectations within a open source community and really just set the ground rules.

Finally as a prediction (and hope), I expect to see more open source communities, foundations and even companies start implementing these code of conduct style guidelines this year.

FYI: Open Source Initiative (@OpenSourceOrg)

If you aren’t aware of the Open Source Initiative (OSI), you should be. They are fantastic not-for-profit organization responsible for the Open Source Definition (which everyone should read once in their lives), they maintain a a list of compliant license definitions on top of promoting open source across the world.

They are also membership driven organization, which is supported by individuals and affiliates. As far as I know, they are the only organization that brings together a variety of open source individuals/institutions to cross-promote ways to work together improve the adoption of open source software:

AffiliateLogosFinal_6

They are also in the last month of their membership drive, so if you’re interested in supporting their cause, I highly recommend you consider joining as a member:

Also more selfishly, the OSI currently has nominations open for the board of director election which I’m partaking in. The current group of nominations include a great group of folks from all over the open source ecosystem and I’d love to have the opportunity to serve, my plans include expanding corporate membership and more.

So please consider supporting the OSI and vote your interests, they really make the greater open source community a better place.

Comments Closed

CFP and Sponsors: MesosCon 2015

We of the MesosCon Program committee recently launched the MesosCon 2015 Call for Papers (CFP) and early bird registration:

If you’re interested in the future of datacenter infrastructure, I highly recommend attending. The conference will be co-located with LinuxCon North America 2015 in beautiful Seattle, WA and the early bird rates are priced at a reasonable $299 to start in my opinion (we also have student rates at $99).

As part of the registration process, you’ll have an opportunity to donate to a MesosCon Diversity Scholarship program which provides support to women, people of color and people with disabilities who may not otherwise have the opportunity to attend for financial reasons. Equal access and diversity are important to MesosCon, and we aim to remove this obstacle for underrepresented attendee groups.

In the coming months, we’ll announce keynotes and the program. We’re also looking for sponsors, so if you’re interested, please feel free to reach out to me.

TravisCI Container Infrastructure: Faster Builds

Just before I disappeared for the holidays, I sent out a tweet talking about testing out the new Travis CI container infrastructure:

Last week, I spent some time moving more @TwitterOSS open source projects on GitHub to take advantage of this and have been nothing but thrilled with the results (seen build speeds improve by 30% to nearly 50%). Faster builds lower the barrier to contribution and also translate into less wasted time.

Anyways, check it out, more people need to know about this rely on Travis CI.

Malicious Open Source Contributions

Yesterday, an interesting happened within the Eclipse Foundation community where someone sent a malicious code review

We generally don’t see this type of thing in open source communities (mostly just contributions without tests), but I believe malicious contributions will continue and become more frequent. The opportunity is just there for bad actors and open source code is embedded all over the place, from your desktop, to your mobile devices to vehicles.

Looking back, there’s been some notable opportunities for bad actors to inject malicious code. One example I recall in particular is RubyGems and SSL and another more prominent example was when the Kernel.org servers were hacked:

Good times, stay diligent.

Apache (and other foundations) considered useful

I couldn’t resist writing a blog about this topic given the chance to use a witty blog title. A few years ago, I blogged about a post that Mike Rogers (@mikeal) wrote about “Apache considered harmful” in the GitHub era.

I agreed with Mike to an extent, but mostly around my frustration in how slow the ASF was in adopting newer tools (like Git) and how the organization was structured with volunteers responsible for critical infrastructure. However, we can save that frustration for another post (note: this has improved as of late).

The interesting part was that Mike recently has had some interesting thoughts about the role of companies in open source due to the NodeJS / io.js forking debacle:

In particular, his opinion is that no company alone can be trusted with the ownership of a community driven open source project. I generally agree with his thoughts however, there are solutions to his problem involving open source foundations. Open source foundations like the ASF, Eclipse Foundation and Linux Foundation (and more) are actually really useful:

The foundations I mentioned above have over a decade of experience being built for the sole purpose of allowing independent open source communities to flourish with fair governance models built on meritocratic behaviors (just take a peak at some of the Apache documentation or Eclipse development process). This is important because the incentives between individuals small companies, large companies, heavily funded companies and even academics are different and need to be accounted for in a fair open source governance structure. Some of these foundations like the Eclipse Foundation started out as the “Eclipse Consortium” and learned some of these lessons the hard way.

In particular, I would like to call out the Eclipse Foundation Working Groups and Linux Foundation Collaborative Projects concepts as some of the best ways to collaborate in the open for maturing open source projects.

On a funny note, as I was trying to get this post out last week, hilariously the container community was going through a fork of Docker with Rocket from CoreOS (in particular, this Hacker News thread was just cheeky):

What happened with Docker/Rocket was almost predictable given the way the Docker project was structured and how late to the game they were in establishing some level of governance and independence as more larger companies were getting involved. At least the competition should help container technology improve at a quicker pace.

In the end, I have to agree with this tweet from Jim Jagielski (@jimjag) about the role of open source foundations:

I hope that in the future as new open source projects become successful, they take a serious look at open source foundations (especially the ones I mentioned) as a proper place to grow and provide structure to their community. Their communities deserve it.

Naming Mars+1 (2016 @EclipseFdn Release)

It’s that time of year that members of the Eclipse Foundation Planning Council help spearhead the community-based naming process of the next Eclipse release (slated for 2016).

The rules are contained in this bug where you can submit names for consideration. Here are the guidelines for names:

The rules and procedure for naming Mars+1 will be similar to what has happened in the past. The name should be alphabetically greater than “M”

Preference will be given to “N” names, but no strict rule that others would not be considered. Preference given to names that fit the “moon”, “heavenly body gods”, or “scientists” themes we’ve had in the past.

I’m suggested Nova or Neutrino to start, but have taken a liking to Neptune as a potential option:

NeptuneHave a better suggestion for a name? Well put it in the bug before we call for an official vote in the coming weeks.

Thank you!