From Buzz to Biz: A Leader’s Guide to Setting Expectations and Scoping First Use Cases

By the end of the decade, AI will be as ubiquitous as the internet is today, touching every department, from IT to human resources. The stakes are high. Those who master AI skills aren't just securing their jobs; they're turbocharging their careers. Imagine a world where proficiency in AI not only guarantees job security but also promises a significant increase in income – up to 35% in fields like IT and marketing.

2023 was all about AI’s promise, 2024 will be all about the proof. We want to make sure you’re equipped with the right evidence and method to scope your first GenAI use case. Let’s make sure you’re on track to reach the potential this new technology has to offer. 

So here we go again…

2–3 years ago the world learned to work remotely/hybrid. A new massive change is among us again… GenAI is re-inventing how we work.

Source: Benedict Evans

So what’s important to know as we integrate our new best friends, these LLMs, into work?

🐣 The sort of obvious

As we navigate the early stages of the Generative AI (GenAI) revolution, it's becoming clear that these technologies offer tangible benefits for enhancing productivity and creativity in the workplace. This is probably the first evidence to consider for your own team when scoping your own use cases. 

Key Insights:

• Productivity Gains: LLMs have shown to reduce the time spent on common writing tasks by 37% and deliver a 40% higher quality of work, as demonstrated by BCG consultants. They also double the speed of decision-making compared to traditional search methods.
• Critical Thinking and Creativity: Beyond mere task completion, LLMs act as cognitive partners, enhancing critical thinking and creativity. This partnership, when leveraged effectively, can lead to the preservation of a creator's authenticity and a heightened sense of authorship.
• Challenges and Pitfalls: However, the journey with GenAI is not without its hurdles. Overreliance on AI, the complexity of prompt engineering, and the nuanced challenges of multilingual performance highlight the need for careful navigation. The criticality of prompt composition and the ongoing challenge in enhancing AI's understanding of non-English and low-resource languages underscore the delicate balance required in deploying these technologies.

 “Minor changes in prompting techniques can yield dramatically different results”

In essence, the obvious with GenAI is focused heavily on productivity gains and improvements in the creative processes. Clear, now let’s have a look at the less obvious insights.

🐓 The less obvious 

As we research further, we’ll find some insights that explain how AI has an impact on tasks and jobs. These might not provide very obvious evidence for your use-cases, but are crucial to consider when trying to scale the impact of GenAI in your organisation. Ready? Let’s dive in.

Distribution of task-load between humans and computers/automation. (Sheridan and Verplank 1978)

Deeper Insights:

• Socio-technical Adoption: Introducing AI into an organisation is a nuanced process that requires considering both social and technical dimensions. It's a collaborative effort where feedback and adaptability play crucial roles in ensuring technology aligns with people's needs and workflows.
• Human Oversight: Despite the push towards automation, the necessity for human monitoring remains. The challenge lies in maintaining attention and making informed interventions, a task that becomes increasingly complex with the introduction of AI. 

Keeping a close eye on things is important, but humans find it challenging to stay focused on monitoring tasks for more than half an hour

• Impact on Jobs and Tasks: While fears of job displacement persist, evidence suggests AI's effect is more nuanced, often augmenting rather than replacing jobs. It's about reshaping tasks and providing tools that elevate the human potential, particularly benefiting those with less experience by levelling the knowledge playing field.
• Challenges in Knowledge Management and Coding: GenAI offers promising solutions for longstanding issues like knowledge fragmentation and can enhance efficiency in coding. However, the effectiveness varies widely, presenting challenges such as prompt misinterpretation and the necessity for sophisticated error handling.

These insights tell us that experiments focused on complex automation, replacing jobs or solving more complex problems in knowledge management or coding are less likely to be effective or successful if you’re just starting out. Focus on the basics, before you continue to experiment with the less obvious.

👽 The surprising

With the basics behind us, let’s have a little fun and look into the future. What surprises can we expect for the workplace? How will they impact our business? The surprising developments not only challenge our conventional understanding of AI's role but also spotlight the emerging skills and innovative approaches that will define the future of work.

Emerging Skills and Insights:

• Orchestration, Synthesis, and Evaluation: As AI enhances information retrieval and content production, the spotlight turns to the integration and critical analysis of information. Skills in research, conceptualization, and emotional intelligence are gaining precedence, marking a shift towards a more nuanced engagement with digital content.
  
  

  Synthesizing, integrating and combining information will become more important skills/capabilities than searching and creating
  

• Metacognition in the AI Era: The ability to reflect on and adapt one's thought processes—metacognition—is becoming crucial. Generative AI tools encourage a deeper awareness of one’s cognitive strategies, improving goal setting, prompt engineering, and the critical evaluation of AI-generated outputs. This introspective skill set allows for more effective collaboration with AI, enhancing decision-making and creative problem-solving.

• From Documents to Dialogues: The transformation of digital knowledge storage from static documents to dynamic, conversational formats represents a significant shift. This evolution facilitates more personal, contextual interactions with information, necessitating advancements in how conversational knowledge is generated, shared, and leveraged for innovation.

• Navigating Uncertainty: The introduction of uncertainty visualization techniques addresses the critical issue of overreliance on AI outputs. By making uncertainty explicit, these techniques improve decision accuracy and task performance, fostering a balanced reliance on AI tools.

  
  UX showing uncertainty in results to improve reliance. Spatharioti et al 2023.
  

• The Pace of AI Interaction: The distinction between “fast” and “slow” AI introduces a paradigm where the speed of response is adapted to the complexity of the task. This approach enhances the depth and quality of AI interactions, enabling more thorough analysis and creative exploration.
  
  

Although the surprising insights tickle our brains for what’s to come, they are not fully fit for use cases or experiments unless you are more advanced in the field of GenAI and rely on staying ahead in the field. For most, sticking to the basics is already better than most. So, how can we turn these insights into practice? Time to design an experiment!

🧪 Designing a GenAI experiment

Designing experiments using GenAI are often focused on internal processes or ways of working. You essentially equip one group of employees with GenAI training, support, and tooling and compare their outputs on a specific set of tasks to a group who doesn’t use GenAI. 

To make sure we’re starting with a proven opportunity in mind, let's consider an experiment inspired by real-world studies, such as those by BCG consultants, who reported a 40% increase in work quality.

• Step 1: Identifying the Problem
  Aim to improve the efficiency and quality of writing tasks within your team.
  
  
• Step 2: Experiment Design
  Group Division: Split your team into two groups. Group A uses ChatGPT with prior training on prompt engineering and ethics in AI, while Group B relies on traditional methods.
  Hypothesis: Group A will complete writing tasks faster and with higher quality than Group B.
  Training: Provide Group A with training on effectively using ChatGPT for writing tasks such as https://platform.openai.com/docs/guides/prompt-engineering 
  Task Assignment: Assign similar writing tasks to both groups, ensuring comparable difficulty and scope.
  
  
• Step 3: Measurement and Evaluation
  Quantitative Measures: Track the time taken and the quantity of output produced by both groups.
  Qualitative Measures: Evaluate the quality of writing using criteria such as creativity, coherence, and adherence to the task requirements. Gather feedback from both groups on their experience through a survey or interview.
  
  
• Step 4: Analysis and Adaptation
  Analyse the outcomes to assess the GenAI tool's impact on writing productivity and quality. Use the feedback to refine strategies for integrating GenAI tools into workflows at a bigger scale, or cover a wider range of tasks.

Want some more inspiration or a more detailed step-by-step guide? Download the toolkit to find more examples for marketing, sales, product, and e-commerce teams!

⚙️ Click here to download the toolkit

The GenAI revolution is set to redefine the workplace by enhancing productivity, fostering creativity, and introducing a new set of skills and challenges. By thoughtfully integrating GenAI tools and adopting structured experimental approaches, leaders can navigate this transformation, unlocking the full potential of their teams and driving forward in the era of AI.