When the Job Description Says AI, but the Data Doesn’t

44.3% of conventional software engineering jobs in our dataset had at least one AI-related skill in their extracted profiles.

That’s a lot of AI for jobs that aren’t explicitly AI engineering roles.

Now, before anyone gets excited, a few things.

Our dataset isn’t the entire job market. These are job records in Glassbox’s current dataset, not a representative survey of every software engineering role. And a skill appearing in an extracted profile doesn’t necessarily mean an employer requires it.

We know. We’re the ones who counted it.

Still, the number raises a useful question: when a software engineering job mentions AI, what does that actually mean you’ll be doing?

First, the numbers

We looked at 6,439 conventional software engineering job records with non-empty descriptions. Of those, 2,851 had at least one of ten selected AI-related skills in their extracted profiles.

That’s 44.3%.

The most frequently extracted mentions included:

• Machine learning: 1,137 jobs
• LLMs: 1,119 jobs
• AI agents: 984 jobs
• AI assistants: 833 jobs
• RAG: 305 jobs

These categories can overlap. A job can mention several of them, and the counts measure extracted skill mentions, not unique jobs requiring each skill.

And here’s the important distinction: a mention isn’t automatically a requirement. It could appear under required skills, preferred skills, or skills merely mentioned in the description.

So we went beyond the aggregate counts.

We read 30 job descriptions

We examined a deliberately balanced sample of 30 descriptions: 15 whose extracted profiles contained at least one of our selected AI-related skills, and 15 whose profiles contained none.

We balanced the groups across backend, frontend, full-stack, and general software engineering roles, and used distinct employers. This is a small, structured sample, not a representative random sample of the entire dataset.

Here’s what stood out.

Among the 15 descriptions whose profiles contained AI-related skills, 8 explicitly discussed using AI tools in engineering work.

Among the 15 with no selected AI-related skills in their profiles, 3 still explicitly discussed using AI tools.

That second finding matters.

A job description can talk about using AI in the engineering workflow without our extraction system recording one of the selected AI-related skills. That could reflect a gap in our vocabulary, a limitation in extraction, or both. We can’t establish the exact cause from these examples alone.

But it shows why we shouldn’t treat an extracted skill profile as a perfect substitute for reading the original description.

Three very different meanings of “AI”

When a software engineering job mentions AI, it can be describing very different kinds of work.

1. Building a product that uses AI

   You might build an application with an AI assistant, integrate an LLM into an existing product, or develop a feature that uses retrieval-augmented generation.

   Here, AI is part of what the product does.

2. Using AI tools to do engineering work

   You might use AI coding assistants to explore unfamiliar code, speed up implementation, improve your development workflow, or help with debugging.

   Here, AI is a tool you use to do the job. It doesn’t necessarily mean you’re building AI systems.

3. Building or operating AI systems

   You might work on model serving, machine learning infrastructure, evaluation, or the systems that make AI applications reliable in production.

   Here, AI systems themselves are the engineering work.

These categories can overlap, but they aren’t interchangeable. A developer who uses an AI coding assistant isn’t automatically an ML engineer. And a developer integrating an LLM into a product isn’t necessarily training models.

The distinction matters when you’re deciding what to learn, which roles to apply for, and how to interpret a long list of skills in a job advert.

What the sample actually showed

In our 30-description sample, the descriptions with AI-related profile skills were not all describing the same kind of work.

Some were about building AI products. Others explicitly described using AI tools in the development workflow. Some had no substantive AI responsibility that we could identify from the description, and a few were unclear.

The group without selected AI-related profile skills also wasn’t entirely free of AI-related work. Three descriptions explicitly discussed using AI tools, despite having none of our selected AI keys in their extracted profiles.

This is why the headline percentage needs context. It tells us how often these selected skills appeared in our extracted profiles. It doesn’t tell us that 44.3% of conventional software engineering jobs are AI jobs, or that every one of those roles requires AI expertise.

That would be a much bigger claim than our data supports.

So, what should a job seeker take away?

Don’t panic when a conventional software engineering role mentions AI. Read the description and look for the actual responsibility.

Ask yourself:

• Are they asking me to build AI-powered features? Look for product integrations, LLM applications, retrieval systems, and related engineering work.
• Are they asking me to use AI tools? Look for descriptions of coding assistants, AI-supported workflows, and developer productivity.
• Are they asking me to build AI infrastructure or models? Look for model serving, ML systems, evaluation, training, and operations.

Then compare those responsibilities with your existing skills and experience. A long list of AI-related terms doesn’t automatically mean you need to become a machine learning researcher before applying.

It does mean you should understand what the employer is actually asking you to do.

What this doesn’t prove

This is a snapshot of Glassbox’s current dataset. It doesn’t establish how these mentions have changed over time, whether AI is replacing particular engineering responsibilities, or whether every employer means the same thing when it mentions AI.

Our sample is also small and deliberately balanced, so its proportions shouldn’t be projected onto the entire job market.

What it does show is a measurement problem worth taking seriously: the words in a job description, the skills extracted from it, and the work a person will actually do are three different things.

A separate working paper by Tom Harris, The New Work of Generative AI: Evidence from 18 Million Duty Statements in US Job Postings, examines millions of US job-posting duty statements over time. It distinguishes new AI-related work activities from familiar tasks performed with new tools. That broader research offers useful context, but it uses a different dataset and methodology; it doesn’t validate or directly explain Glassbox’s percentages.

The point isn’t to count AI mentions. It’s to understand the work.

A job description is an account of expected work. A skill profile is our interpretation of that account.

They’re not the same thing.

At Glassbox, we’re trying to make the technology job market easier to understand by counting what employers ask for and showing how those requirements relate to your skills.

We show, you decide.

Explore the job market at seeglassbox.com.