LLMs affecting developers’ work & interactions

Research Question

With the proliferation of AI tools and Large Language Models (LLMs) in most fields of work and for the most diverse tasks, we wanted to research and understand how these tools’ sudden expansion could be perceived and affect workers’ job and their interactions with colleagues.

In particular, we were interested in learning more about how LLMs could interfere with the work of developers and computer scientists and how they deal with issues and uncertainties now that these tools are available, potentially with AI taking over their relations with colleagues when it comes to asking for feedback or help.

The process

The research began with virtual observation to capture real-world behaviour in an unbiased context, avoiding reliance on participants’ retrospective accounts and interference from our side.
This was followed by interviews, which allowed for deeper insights and detailed exploration of specific aspects noted during observation.
Lastly, a focus group and a workshop encouraged collective discussion, enabling new topics to emerge through group dynamics and offering participants diverse ways to express their thoughts.

Initial Observations

To gain insight into users’ current usage and perspectives on these technologies, we conducted 15 virtual observations. This approach allowed us to gather data in a naturalistic context, without directly interacting or interfering with the discussions developers initiated on forums and websites.

Through this method, we aimed to capture comprehensive real-world information, identifying recurring themes to be coded using open and, subsequently, axial coding techniques.

Interviews

Based on the themes that emerged from our observations, we used the interview phase to gain a deeper understanding of developers’ key behaviours and perspectives.

Participants were selected through a convenience sample, directly inviting them or through word of mouth and snowball sampling.
A total of 6 interviews were conducted with people at different stages and levels of expertise.

Interview structure and questions

• [Greeting & Formalities]
  • Greetings and general information
  • present the research
  • Informed consent
• [Basic Information]
  • General information (Age, Gender, Years of experience, Education level)
  • Job Description (“can you also introduce your work environment and your job?“)
  • work routine (“What is your daily working routine like?” “Is there anything you do(n’t) like about it?”)
• [Main body of interview]
  • Managing problems or errors arising during the job (“How do you manage to deal with these [problems], are there any example situations you can come up with and tell me?”)
  • Exploring use/not use of LLM (ChatGPT) to solve problems (“can you explain a little bit more about how do you use it? What do you think of it? What tasks do you usually ask at chatGPT?” “is there any reason to not use ChatGPT? would you use them and in which situations?”)
  • Incurring into problems using ChatGPT
  • Thoughts about ChatGPT output response quality
  • Feeling about availability of ChapGPT vs colleagues (“How do you feel about AI tools being always available in comparison to traditional tools, your colleagues or or other way to obtain information and help?”)
  • Feelings about answers got from ChatGPT vs colleagues (“What are the feelings you get when receiving an advice or a critic? how does it feels when you do the same with an AI tool?”)
• [Discuss about Future impact]
  • Impact on future education
  • Impact on future work environment & developer position
  • Potential changes in skills & requirements for developers
• [Conclusion]
  • Space for additional comments from the interviewee
  • Debriefing & Final thanks

Participatory Design Workshop

Take Aways & Results

From the various sources of insight we applied during our research, we collected some recurring and well-established information about our participants’ points of view and feelings about the emergence of this new technology and the effects it has now and may have in the future, on the developers’ environment.
More precisely, we can affirm: