AI Agents in Construction: What They Are, Where They Fit and What to Ask
By Kush Agarwal, President and CEO, YelloSKYE. Published .
Executive Summary
- The Question: What are AI agents, what could they realistically do with construction site data, and how should a project team decide whether to trust one?
- The Answer: An agent is software that takes a series of actions towards a goal instead of just answering a question. On site data, the realistic early uses are preparing evidence packs, chasing open issues and drafting updates, with a person approving each action. The risks are errors repeated at speed, poor data and unclear accountability.
- The YelloSKYE Angle: We do not sell agents. We produce the located, measured and reviewed site record that any software, agent or otherwise, would have to rely on, through Construct IQ and the DEX platform.
An AI agent is software that is given a goal and then plans and carries out several steps to reach it: it reads data, uses tools, checks what happened and decides on the next step, often without a person directing each one. That is the key difference from a chat assistant, which answers a question and stops. In construction, agents are an emerging industry development rather than an established tool, and their usefulness on site data depends almost entirely on how good that data is. YelloSKYE does not offer agents; our part is the site record itself, captured and organised through Construct IQ, our construction monitoring solution.
This guide explains what agents are in plain terms, where they could realistically help on construction site data, the risks, and the questions to ask before you let one act on a project. For the wider picture of AI on projects, start with our guide to AI in construction.
Assistant, automation, agent: three different things
Vendors use "AI" and "agent" loosely, so it helps to separate three kinds of software that are often described with the same words.
| Kind of software | What it does | Who decides each step | A construction example |
|---|---|---|---|
| Assistant | Answers a question or drafts text when asked | The person, every time | "Summarise the RFIs that mention grid line 7" |
| Rule-based automation | Runs a fixed sequence when a set condition is met | The person who wrote the rule, in advance | "When a milestone is reached, notify the project manager" |
| Agent | Works out a sequence of actions to reach a goal and adjusts as it goes | The software, within limits a person has set | "Prepare this month's evidence pack for the earthworks claim" |
The line between the last two matters. Automation does the same thing every time and fails in predictable ways. An agent chooses its own steps, which makes it more flexible and also less predictable. A useful test: if you cannot write down in advance every step the software will take, you are dealing with an agent.
How an agent works
Under the marketing, most agents share the same parts.
A goal
The task is stated in plain language, such as "find every open snag on level four that the site record suggests has been fixed". The quality of the result depends heavily on how precisely the goal is stated.
Tools
An agent can only do what it has been connected to. Typical tools are reading from a document store or a site record, searching images by location and date, creating or updating issues, writing a draft report and sending a notification. Each tool is a permission, and each permission is a risk to manage.
A loop
The agent plans a step, carries it out, looks at the result and decides what to do next. It might pull a list of open issues, fetch the latest images for each location, compare them, then draft an update. If a step fails or returns something unexpected, it tries another route.
Limits
Good agents run inside boundaries set by people: which tools they may use, which projects they may touch, which actions need approval and when to stop and ask. Those limits, more than the underlying model, decide whether an agent is safe to use.
Realistic uses on construction site data
Most early agent work in the industry has been on documents and email, where data is already text. Site data is harder, because the agent needs to know where each image was taken, when, and what the design says should be there. Where that record exists, these are the kinds of tasks where an agent could plausibly help, each with a person approving the output.
| Task | What the agent would read | What it would produce | What a person still does |
|---|---|---|---|
| Payment claim evidence pack | Claimed progress, measured quantities, dated site images | A pack matching each claimed item to evidence | Decides what to certify |
| Open-issue follow-up | Open issues, their locations, the latest captures there | A list of issues that look resolved or still open, with images | Closes or reopens each one |
| Pre-closure check | Programme dates, 360° records of areas about to be closed | A list of rooms with no recent capture before boarding | Arranges a walk or holds the closure |
| Delivery reconciliation | Delivery notes, stockpile volumes from surveys | Gaps between what was delivered and what is measured | Investigates the difference |
| Owner update draft | Progress by area, recent views, milestone dates | A draft update with images attached | Edits and sends it |
| Handover pack | Room-by-room records from fit-out | An organised record of each space before closure | Checks it is complete |
Why these and not others
Each of these tasks is mostly gathering and matching: finding the right records, putting them side by side and spotting gaps. That is the kind of work people find tedious and agents can do quickly. None of them asks the agent to accept work, certify payment or instruct a contractor, which is where errors carry real cost.
Tasks that involve judging safety, workmanship or blame are a different matter. An agent can be asked to look for a particular visible condition in a set of images, but it cannot see context, intent or anything out of frame. Treat any output of that kind as a prompt for a person to look, not as a finding.
Why site data is a harder test than documents
A language model can read a specification directly. It cannot read a site in the same way. For an agent to work on site data, someone has to have already done the groundwork:
- Every capture placed in space, on the site, on the drawings and in the model. Our guide to spatial AI in construction explains why this is the foundation.
- Captures repeated over time, from the same routes and viewpoints, so "has this changed?" has an answer.
- Design and programme data that is current, because an agent comparing against a superseded model will be confidently wrong.
- Measurements with known accuracy, so a quantity can be relied on in a claim.
Without these, an agent has little to work with beyond loose photos and text. With them, much of the hard work is already done before any agent is involved. Our guide to visual intelligence in construction describes what that organised record looks like.
The risks
Agents bring the usual risks of AI, plus some of their own that come from acting rather than advising.
| Risk | How it shows up | How to reduce it |
|---|---|---|
| Errors at speed | One wrong assumption repeated across hundreds of records | Start with small batches and check a sample of every run |
| Acting on poor data | Issues closed because an image of the wrong room looked tidy | Require location and date checks before any action |
| Permissions too wide | An agent able to send messages to subcontractors it should not contact | Grant only the tools a task needs, project by project |
| Unclear accountability | Nobody can say who approved an action | Name the person responsible for each agent and each approval |
| Silent failure | The agent stops working and nobody notices | Report every run, including runs with nothing to report |
| Data sharing | Project data sent to a model provider without agreement | Check where data is processed and whether it trains other models |
| No audit trail | An action cannot be explained in a dispute | Log every step, input and output, and keep the log |
Accountability deserves emphasis. Contracts recognise people and companies, not software. If an agent closes an issue that was not fixed, or drafts a claim with the wrong quantity, the responsibility sits with whoever let it act.
How much autonomy to allow
A practical way to manage agents is to decide, task by task, how far along this ladder the software may go.
| Level | What the agent may do | Suitable construction tasks |
|---|---|---|
| Suggest | Point out items for a person to look at | Possible safety or quality conditions in images |
| Draft | Prepare a document or list for approval | Evidence packs, owner updates, issue follow-ups |
| Act with approval | Carry out an action after a person confirms it | Closing issues, sending routine reminders |
| Act and report | Carry out low-stakes actions and log them | Filing records, tagging images, scheduling reminders |
Most construction tasks that touch money, safety or contractual position should stay at the first two levels for now. Move a task up only after the agent has been right consistently on your own projects, and be ready to move it back down.
Questions to ask before trusting one
Whether the agent comes from a vendor or is built in-house, these questions separate a credible tool from a demo.
- Exactly which actions can it take? Get the full list of tools and permissions, not a description.
- What data does it read, and how current is it? An agent is limited by its inputs.
- How does it show its evidence? Every output should link to the records it was based on.
- What does it do when it is unsure? It should stop and ask, not guess.
- Who approves its actions, and can that be enforced? Approval should be built in, not optional.
- Is every step logged? You need to be able to reconstruct what it did and why.
- How was it tested, and on what kind of project? Ask for results on work similar to yours.
- Where is the data processed, and who owns the outputs? Settle this before any project data is connected.
- What does it cost to run? Agents that run often can add usage costs on top of a licence.
Where YelloSKYE fits
YelloSKYE does not offer AI agents. What we do is produce the kind of site record that agents, and the people using them, would need.
Our pilots or the client's site team capture the site with drones, 360° cameras, fixed cameras or laser scanners. DEX aligns those captures with CAD and BIM, flags deviations, calculates volumes and keeps the progress dashboard current. Construct IQ compares progress against the schedule baseline in 4D, sends alerts when critical milestones are reached or activity stops in high-priority zones, and lets teams tag issues on the site view and assign them. Our civil and geospatial engineers review processed outputs before they reach a dashboard. Exports to Procore, Autodesk Construction Cloud and MS Project keep that progress data next to the systems a team already runs.
Three use cases show the kind of record this produces. In resolving contractor billing disputes, drone captures aligned with billing cycles check claimed progress against the site. In earthwork volumetric analytics, comparing terrain models with the design brought a two-week survey cycle down to a few hours. In portfolio monitoring with DEX, progress across many sites is brought into one dashboard. For a step-by-step view of how one day's captures become a reviewed record, see what AI does on a jobsite.
A sensible way to start
- Get the site record right first. Regular, located, dated captures and current design data are the precondition for any agent on site data.
- Pick one gathering-and-matching task, such as an evidence pack for a monthly claim, not a task that requires judgement.
- Keep it at draft level. Let the agent prepare; let a named person approve.
- Limit its tools and projects to what the pilot needs.
- Check a sample of every run and record how often it was right.
- Agree data terms in writing before connecting project data.
- Expand only on evidence, one task and one level of autonomy at a time.
Frequently asked questions
What is an AI agent in construction?
An AI agent is software that is given a goal and then works out and carries out several steps to reach it: reading project data, using tools, checking results and deciding what to do next. The difference from a chat assistant is that it acts, for example by drafting records, updating a list or sending a message, rather than only answering a question.
Are AI agents being used on construction sites today?
They are an emerging development. Several software vendors and some contractors are testing agents that work on documents, schedules and site imagery, mostly with a person approving each action. Fully unsupervised agents on live projects are still uncommon, and most teams are at the pilot stage.
What are the risks of AI agents in construction?
The main risks are confident mistakes repeated at scale, actions taken on poor or out-of-date data, permissions that are wider than needed, unclear accountability when something goes wrong and project data shared in ways nobody agreed. Each can be reduced by limiting what the agent may do and keeping a full log of its actions.
Does YelloSKYE offer AI agents?
No. YelloSKYE captures sites with drones, 360° cameras, fixed cameras and laser scanners, and DEX places, aligns, measures and flags that data, with engineers reviewing the output. Any agent a client adopts would depend on site data of that kind being accurate, located and current.
What should we ask a vendor before trusting an AI agent?
Ask exactly which actions it can take, what data it reads, how it shows its reasoning and evidence, what happens when it is unsure, who approves its actions, how every step is logged and how it was tested on projects like yours. If the answers are vague, keep it in a suggest-only role.