What do VPs of engineering care about

Last week we had our first Engineering Leaders Forum, here in the Bay Area. Over 100 VPs and SVPs of Engineering from the Bay Area’s leading companies registered. At some point we had to open up a waiting list. Good problem to have. 

I have wanted to hold such an event for quite some time. Engineering leaders, as opposed to other personas, lack the opportunity to share their challenges and hear the perspective of others in a very safe environment. In the room last week we had a few thousands of years of experience. Based on the feedback, the format of roundtables was well received. I really appreciate the fact that both Kubiya.ai as well as Devzero, Turing, Uplevel and AWS chose to sponsor the event and make it happen. I want to talk a little bit about some of the insights that we had from executives in the room. Again, I'll keep it generic so that I wouldn't share anything confidential.

Two key items that we heard from all the executives in the room, one was centered around AI and the other one was around measurement.

What do Engineering leaders think about AI and ChatGPT?

The feeling is that we're in an interesting era. The same transformation that happened 10, 15 years ago with the cloud is happening today with Artificial Intelligence (AI). Everyone wants to “do something” around AI. I also hear that from our customers and prospects. But many are still figuring out what exactly they want to do around AI and what are the use cases. We heard lots of concerns about data privacy. Members mentioned that with tools like ChatGPT, they don't really know how data is stored, what is stored and who might have access to it. One mentions that the same way his company can gain an edge by using AI, he can also give an edge to their competitors using their data. Most leaders are taking baby steps towards using AI. A few companies in the room mentioned actually trying various things around AI, using Copilot or ChatGPT, to generate code for example. 

The other thing that we heard very strongly and we ourselves are in the midst of it so we weren't surprised with that, was the fact that AI today is very generic and lacks the contextual organization. And so as opposed to giving generic code snippets, if AI could look into the organizational repo and make more contextual recommendations, or the case of Kubiya, if AI could actually figure out the engineering resources and platforms that an organization is using, and be able to spin up environments based on that, or render Terraform models, that will be very powerful.

Early adopters mentioned that they're using AI mostly for code reviews. Developers find code reviews to be unproductive, so having another set of “eyes” can expedite the process and remove dependencies.

Again, in some ways it feels like AI is a disruptor and everybody wants to be in the game and use it to gain an edge, but the use cases are still not there and data privacy is a big issue.

One additional item that was discussed was the use of ChatGPT inside organizations. On the one hand, unless they block the domain, organizations have no control over how people in the organization use ChatGPT. You can come up with policies to restrict the use of it but it’s difficult to enforce that. If one uses ChatGPT to write code or create a document you are exposed to sharing private information or violating licensing rights. Organizations don't have great policies right now nor giving clarity to employees around when to use AI and ChatGPT. One last thing that was mentioned specifically around tools like Copilot and other AI code generating tools is the fact that given that code snippets were used to train their LMM, the output at some point might violate copyrights and will expose them to legal actions.

How do engineering leaders measure their teams?

We had a number of roundtables on scaling engineering organizations and increasing velocity. My colleague Debo Ray from Devzero wrote about it in “Impact of AI, Product Velocity, and more: Learnings from Engineering Leaders Forum”. At the end of the day the discussions quickly came back to the theme of how do you measure engineering organizations? Some of the leaders in the room weren't as familiar with DORA as others. And so it really feels like managing and measuring an engineering organization is still as much as art as it is science. One of the leaders mentioned around the fact that he's looking at the number of commits per week, per developer as a measurement, and that together with some context around whether that engineer was sick, busy in interviews, in a conference or doing something else, over time gives them good visibility to how the team is performing. 

Remote work and RIFs

Prior to the roundtables we had a chance to talk to members 1-on-1. It quickly came down to two topics. Layoffs and remote work. Many of the leaders had to reorganize their teams and let go over good people. Leaders these days are more focused on efficiency than scale.

It was interesting to hear the views around remote work. 12-18 months ago companies were talking about remote work being the new norm. Most members were scaling back the transition to remote work, citing the overhead of managing remote teams, lack of visibility and the impact on culture as being the key reasons they ask developers to work from the office 2-3 days a week. Especially with layoffs, culture and communication becomes important and it’s easier to communicate and build culture with teams being onsite. Remote work also seems to impact engineering leaders themselves. Working remote from your home office seems less favorite than it used to be. We work longer hours, don’t socialize enough and lack better work-life balance.

Summary 

Members really enjoyed the format and the opportunity to share knowledge and experience with one another. The discussions were frank and transparent and the combination of in-person 1-on-1 conversations with a 10-12 ppl roundtable offered lots of insights and food for thought.

See you on the next ELF.