Praqma is at the spot (called the Egg) in Brussels, when GitHub invites Gitters and geeks from all of Europe to come together in 2 full days, where the entire world turns around just Git. We will be blogging from the scene for the next two days.
An Unexpected Party
Two years ago, a bunch of Praqma colleagues went to Git Merge in Paris. In many ways that event came to be a bit of a game changer for Praqma; It was at Git Merge Paris in 2015 we had our first interview with Andrey, who short time after ended up kick-starting our Stockholm office.
It’s where we first heard Ed Thomson speak. A brilliant speaker we later invited to speak at our own conferences in both 2015 and 2016 (and who since then actually joined GitHub). It’s also where we first met Ewelina , a Polish DevOps’er who later attended our CoDe Academy in Copenhagen and since January 1st this year is relocated to Copenhagen and now employed with us.
Back then we also weren’t partners with GitHub, that happened later. Back then Praqma was 12-14 people, today we will be turning 40 with our next employee.
It’s kinda like we see Praqma as before and after Git Merge 2015.
So obviously we’re here again - at the GitHub party - when the opportunity reveals itself.
Over Hill and Under Hill
It was a quite bumpy road here. It turned out, the day before we were set to go, that neither Jasmine nor myself actually had tickets for the event. In the heat of it all, with all the logistics, we arranged flights, speaks, accommodations. But we forgot to go the Git Merge web site and buy tickets.
And now it was sold out - Ooops!
Fortunately we have a direct line to GitHub who is organizing the event. Our red phone is really an organization on GitHub, that we are sharing with the Githubbers, normally used to work together on GitHub issues, but at this point I cried for help.
Less than an hour later, everything was sorted out. Our GitHub friends had set the earth in motion and rocked their colleagues who were in Brussels, setting everything up, and when we arrived this morning we were greeted with hugs and full access conference pass to the whole shebang.
Githubbers are nice!
Next up is Jan and Johan! - our homies!
Riddles In The Dark
Jan and Johan as they enter the stage.
Johan is giving an excellent 5 min introduction to Git hooks. Then he shows how Git can be turned around and become fully policy enforcing by implementation of hooks; prevent commits on master, check if messages references an issue, require fast-forward only merges to master, run tests before pushing to origin,
Trust your workflow - fear is the path to the dark side
Essentially he is enforcing the phlow - using local hooks.
Half way through the 50 min presentation, Jan takes over - presenting an analogy to Tantooine - a planet portrayed in The Star Wars movies; a lawless dessert planet, ruled by Hutt gangsters.
Taking us through git diff tricks - swapping default diff’ers in
.gitattributes with some quite unorthodox alternatives - with surprisingly useful side-effects.
But hey! Encouraging people to store binary files in Git by making it easyand useful! Maybe it is a step towards the dark side?
Jan also shows how automated on-the-fly encryption allows users to store secrets - in public - on GitHub.
Hmmm - bending rules is efficient! Dark is good - let that be a tribute to Leonard Cohen’s last hit “You Want It Darker”
- “Will this session become available online?”
The crowd is impressed - there is spontaneous applauds through out the presentation even a few airy “Wooow!”.
GitHub is recording the session - we’ll dump a link here once it’s online. Until that happens, you can get the slide deck.
Jan presenting an image of Yoda - diff’ed in ASCII.
A Thief In The Night
I’m not a contributor to the git core.
But after the lunch, which only added to the feeling that this is a conference with a far larger budget than the entrance fees from the participants, I snug in - like a thief in the night - to the Git Contributor Summit.
Yes! This is where the people that drive the manifestation of the modern software development revolution is gathered to debate, conspire, share and work together.
Benevolent dictators and trusted lieutenants, even foot soldiers and reviewers are here.
The whole setup is run like as an informal unconference; Participants add topics they want to discuss to a flip chart. Then they vote on what is most interesting or urgent, and then they discuss the top-most voted issue for about half hour, based on a lightning presentation from the person who put it there to begin with.
Google, Microsoft, GitHub… - the thought-leaders all sent their evangelists to this gathering of the Git intelligentsia.
As I arrive as Johannes Schindelin @dscho is kicking off a discussion on the tool stack that Git contributors are using. Johannes is another source of great inspiration, whom I also met first time two years ago at Git Merge in Paris. Today he’s working for Microsoft and driving their initiative on supporting Git in VSTS.
I chatted with Johannes in the break, he let me in on a secret; Microsoft is working on loosing the dependency to libgit2 in VSTS and go all git native. The main reason is that git in general is more up to date with new desirable features - LFS for starters. Johannes rounds up, as he’s being called on to present the next topic; “actually it’s not a secret anymore it’s now public knowledge”.
The tool stack discussion soon evolves into a workflow discussion. “Junio” is mentioned again and again. Junio is the Benevolent Dictator of Git, a leadership he was personally granted by Linus himself. Junio is not here today, but it almost feels as if he was. Despite the fact that he clearly is in the top of this hierarchy, everyone speak of him respectfully and the crowd here it trying to come to some kind of agreement, so they can jointly approach Junio with a serious proposal of improvement.
The Git intelligensia gatherering at Git Merge
Everyone listens when someone speaks, good arguments are respected. No one is agitated or dominant. There’s a sentiment of seriousness in this room. This is clearly the meritocracy at work. It might not be the most efficient way of working - in terms of pace, but again we’re talking about the git core - probably one of the most important pieces of software - in the world.
With this process, it’s not likely that serious mistakes are going to sneak into the core. I feel safe - and thankful - that there are companies out there, who pays these guy’s salaries, so they can find the time to dive into these details.
That being said, I have to admit, that the process for contributing to git which is based on mailing in patches generated with the gitdiff tools to the mailing list. It seems to be rather hmmm old-school and could potentially be optimized quite a bit by introducing forks, branches and annotated reviews in the process. Simplicity provided by the phlow, which is git centric and doesn’t utilize any proprietary developed features that aren’t already available in git core, and which is specifically designed to support a benevolent dictator governance model around a release train branching strategy - could really come in handy here.
But this is not the place, where the thief, that snug in, will come forward and make suggestions.
Next topic up is on storing key/value tuples securely - I’ll just sit tight and listen in on that too.
A Warm Welcome
Today is the big day. Gone are yesterday’s workshop and contributor summit. The second day of Git Merge is a one track conference. No breakouts - no fuzz - one track only.
It’s like going to a gourmet Michelin starred restaurant; there’s no a la carte menu. Instead you have the best chefs, in terms of event organizers, and they have composed a delicious 12 course menu - and that is what you are having.
But before we start with the program we need to get through the breakfast, a lot of Belgium style desserts and cakes. I’ll skip the toast - and go straight for desserts.
Breakfast is served at Git Merge.
Karen is a cyborg - her heart is three times the size of a normal sized heart, so she has a defibrillator implanted.
When she was given this device, she was more worried about what software the defibrillator ran on. Everyone know that all software is potentially buggy. So she tried to get to see the software on the device before the implant, but se was refused, the software was proprietary.
Karen obviously had to take the defibrillator anyway - but it was the the kick-off for starting the Software Freedom Conservancy.
The conservancy does legal work, handle foundations and fundraising. They give advise on licensing and trademark protections. They even helped the Git community.
GitHub is turning over the revenue of this particular Git Merge conference to Software Freedom Conservancy - it adds to a total of 22K USD.
Karens defibrillator is a metaphor for all the software out there that we rely on - as she put is:
We’re only as free as the software that we use.
Let’s set software free!
Karen Shandler - the big hearted cyborg, and executive director of the Software Freedom Conservancy - at Git Merge
Ed Thomson goes on stage for a couple of minutes and gives some insight into the git contributor summit that took place yesterday, and explains how it works - but hey, I already covered that in this blog yesterday.
He’s immidiately followed by Carlos Martin Nieto, infrastructure engineer at GitHub. Carlos give insights into some of the challenges that GitHub meets when hosting their 14 million user infrastructure; “Top Ten Worst Repositories to host on GitHub”.
It’s a very hmmm personal presentation, Carlos is taking us through the top ten worst repos. It’s a bit of a dropping numbers presentation, but surely numbers can be quite amazing in the right context. An everyday example is a community that is organizing commit wars; thousands of users committing to the same repo, at the same time, so see who gets in. GitHub Ops tried to convince them not to go ahead with it, but good ideas don’t stop for no one, so obviously GitHub was left with a challenge.
Some of the repos he mentions are just personal user’s repos that represent funny (read: obscure) user scenarios. Others are just large networks; Did you know that the Linus’s Linux core is forked 16K times? octocat/Spoon-Knife is GitHubs sample repo, used to illustrate what forking is all about - it’s forked 88K times!
Carlos wraps up:
You and your users grow up together, it’s fine. Everyday is different - and my job is fun
Now, that’s the attitude - thanks Carlos!
Barrels out of Bond
Saeed Noursalehi from Microsoft is now taking the stage to talk about scaling Git at Microsoft with more that 30K developers.
When they started with the humongous Windows repository it would literally take days to clone the repository. Git is fast - but 8 mins to get a nothing has changed status seems very excessive.
Again we’re being presented with the case that Git’s assumptions generally work well for reasonable repositories - but where is the fun in that?
They’ve now implemented GVFS - Git Virtual File System - Git LFS did not change the size of the index - so that was not a helpful avenue to go down.
Microsoft using GVFS - numbers in the realm of the acceptable
Microsoft is open sourcing a lot of it and publishing it in a few minutes - as we’re sitting here! That’s an awesome worlds first!
We’re getting in deep, it is basically a file system driver combined with a read-object hook in Git. The read-object hook is a new concept that Microsoft has introduced to Git - seems like we can do even more crazy stuff that we did in our talk yesterday!
Saeed is a brave man - he’s starting on live demo on a VSO - the demo gods punished him with brief technical issues, but “When in doubt you just reboot it, right?”
Luckily Microsoft also cares about other platforms the message is “We’re hiring - especially if you know file systems on Mac and Linux” - Hmmm let’s use this occasion to mention that “If you know Continuous Delivery - then Praqma is hiring too”.
The trees doesn’t grow in to the skies - when asked about searching globally in your repo - answer was “Yes, that will still be very painful”.
Out of the frying pan - into the fire
Next up: “What is wrong with Git?” - now the truth is brought by Santiago Perez de Rosso.
According to Santiago, Git is hard to learn and the documentation is scary. Good that we have such a strong trainer corp at Praqma.
Santiago is joking and provoking with a slide about how commits can easily be represented as a manifold in a multi dimensional space.
In his Phd work, Santiago has been surveying user response to the usability of git, and experimenting with fundamental improvements to the git experience. Now he is discussing the complex understanding of concepts like stash, detached head and untracked files.
Santiago Perez de Rosso at Git Merge, comparint Git to Gitless
In his experimental git alternative, Gitless, a branch includes it’s working dir, so dirty files never prevent a branch switch. Other similar changes were made to the other confusing topics. Gitless was then used to run parallel experiments with non-expert users to compare how learning Gitless compared to Git.
Gitless was just an experiment, but Santiago suggests that the world might be ready for new VCSs, or at least a new porcelain level on top of Git, that look very different than Git, but builds upon the same underlying plumbing.
Gitless is available for download and experimenting if you are curious.
A Short Rest
We’ve had a full lunch, again challenging the delivery pipeline of the food people. We’re back in the auditorium - now Caren Garcia is sharing her experiences teaching Git.
According to Caren, Git conflicts induces panic states in students. The command order confuses them, but they love the utility of having everything in Git.
Collaboration is both one of the best and worst things with Git. Don’t we know it - most interesting things happen in multi-user scenarios.
Caren recommends the web site OhShitGit the get help to get out of panic situations.
Caren is mirroring our own strategy in teaching Git - Focus on basics and lot’s of practice! She also wants everyone to use Git - Writers, Governments and school.
A good place to mention that this blog is written in MarkDown, committed to git, build with a pipeline hosted on Jenkins, including Jekyll and a few more verifications and published and served on GitHub pages.
How amazing would it be to do git diff on a law change?!?
We’re glad to hear that her students has had great success using Git CLI for teaching, this is something that we very much also experience in our own training. Don’t try to hide the details - teach the details!
Not At Home
Durham Goode is now taking the stage with his award winning slides. He is going to talk about how Mercurial (Hg) has been scaled at Facebook.
For one, they have implemented
PushRebase, which basically allows the server to perform your rebase for you in some conditions. It looks really cool. Git seems to be close to be able to implement this - we just need to be able to return a commit during a push.
Durham reveals that, in order to overcome their tremendous size “Everyone commits directly on master.”
- “Hey - What is going on?”
It seems like a root-cause analysis is called for as much as new features.
Anyway, they tend to assume their developers are online -
InifinityPush - assures that every commit ever is pushed - This sounds like a hybrid between Git and ClearCase.
A topic that comes up again is the User Experience of Git. The way Git log is working seems very useless from Durham’s perspective.
Durham asks: “Why isn’t the default behavior of the git log useful? - Why do we all have to go through hoops to customize the log enough to make it useful?”
They have made a
smartlog, that allows the to recover from for instance commits that’s not reachable on any branches.
At the end, the critical crowd asked “Why isn’t this Open Source” to which the one acceptable answer came… “It is!” Facebook runs on the latest stable Mercurial build and hereby catches a lot of Mercurial bugs before they hit others. Not everything is contributed back to hg-core, mostly due to the fact that not all contributions the make are backwards compatibility.
All in all the take-away from Durham Goode was that the Git community has the power to impact the workflows of an entire generation of developers - and the best way to do that, is to start focus on the UX of Git. Enable new users to do power user moves. He uses
hg unamend and
hg uncommit as a few good examples.
A good motivational speak from the other side of the fence.
Back in the days, when I used Hg extensively - I found myself utilize the
addremove command a lot.
I still do in git:
[alias] addremove = add -A .
Git may not have the coolest UX - but it is tweakable.
Now off to a brief break where the people will be hydrated and the batteries recharged.
The gathering of the clouds
Just in case we were jealous that we were not attending the Git contributors summit, Lars Schneider and Taylor Blau runs us through how to contribute to git core.
It’s not just about the feature, it needs to be documented, and tested as well.
In case you didn’t know the Git community really cares about commit messages
In general a commit message should be self-contained, meaning you need no external decencies to figure out what the commit does. I guess this is what happens when you are not using an issue tracker.
So many steps that you have to go through and formalities to be respected to be able to contribute to Git core - a bit scary!
As an example of the friction in this approach, Lars gives us the example where he added a very hefty performance improvement and it took 6 months and 380 emails for him to get it approved.
Git is used by millions all over the world, you don’t want to be the one breaking it.
Taylor now take over and talk a bit about Git LFS.
Using the Scanner pattern in GO was a perfect match for the way Git plumbing passes information to Git-LFS
Taylor up out by sharing that the improvement they’ve measured on average is 80 times faster for repositories with Git-LFS enabled.
He shares the next steps: Promise Capability, File-path exchange, and work on figuring out how to get the metrics to make git LFS even faster.
Tim Petterson from Atlassian now takes over the stage to talk about “Git aliases for Gods”.
It is not just about saving time and keystrokes - when ever you find yourself googling the same thing over and over - you should consider make an alias.
Tim takes of through the four different ways that you can stash, and how that interacts with the index and working directory. He made aliases; the more
as you have in your command the more you stash:
staaash. - he he.
Bitbucket has joined the CD world for real, emulating a lot of what services like Gitlab and travis are doing - they have enabled pipeline in bitbucket - to allow you to build any branch.
We see an awesome, and very silly way, to spoof git sha’s with the help of emojis and raw processing power.
Tim expertly finishes by combining Bitbucket Pipeline with an alias, so he can do a
git bisect while utilizing the Bitbucket pipeline - he also just runs all the steps in the bisect parallel - that’s bruteforcing.
We agree with Tim that CPU time is worth less than dev time.
The Last Stage
The last part of the program is opened with 3 lightning talks
Tamir Gefen - is presenting a ClearCase case story, moving out of ClearCase, based on adjusting ClearCase views one at a time and then dumping content on new branches in git. In all humbleness, allow me to point to the Open Source project 2git.io as somewhat more sophisticated alternative.
Meredith L. Patterson is showing us how git chosen as the underlying platform for implementing a moneyless transaction exchange ledge system, implemented as an actual monetary system in Sierra Leone, where phone minutes is actually considered a viable FX currency, since otherwise, the national bank would go bankrupt.
Sometimes the would is crazy - Send more money.
Santiago M. Mola presents a pure Go Lang implementation of Git.
In “Git for Pair Programming” Cornelius Schumache is taking us through the basics in paired programming. Mostly a “Hey!” for paired programming as a discipline, not very git related.
But touches on git-duet, which utilizes
author to show the identity of both contributors in the pair.
Not useful for mob programming though.
Cornelius goes into details with the multiple authors problem using anything from emails, commit messages and headers…
Allow me to suggest: Tie every commit to an issue, add more contributors to the issue the phlow would support that.
But hey, hands down: Paired Programming is cool and fun - there’s always room for you another “Hey!” on this topic.
The Return Journey
Now it’s Saturday morning. Yesterday evening we all attended the Git Merge Conference party, held at a party venue with attitude; an old brewery rebuilt into a club, with a very loud sound system and a DJ, who ruled the dance floor - so loud, that the entire party fled into a side room, where a chocolate taster booth and a beer taster booth was completing for everyones attention. While everyone essentially just wanted to turn the music down and have geeky conversations with their geek peers.
Yup, geeks know how to party.
My travel companion Jan chose to extend his stay in Brussels to attend FOSDEM, the rest of us is heading home. I had set my alarm clock, enabling my usual everyday alarm, to go off at 6:50. Hmm, I forgot that I don’t usually get up at 6:50 on Saturdays - and today is Saturday - so my alarm didn’t go off - exactly as it shouldn’t.
Luckily Johan and Jasmine was up early and they could kick my door in and get me up.
Now we’re in the flight home, it’s Git Merge 2017, wrap-up evaluation time.
Personally, meeting a lot of old friends and experiencing the professionalism of the GitHub staff and how they organized the event - and scheming and planning together with Jasmine, our own event manager, who’s responsible for the 20+ events that Praqma is running annually ourselves was the best takeaway for me.
The insight in to how the Git contributors work and organize was also important. We have been engaged in the Jenkins Community for many years, and we’re facilitating our own Open Source in the Continuous Delivery community CoDe Alliance.
Cheers everyone - see you out there - AFK - at some other event.