Original ImadDhin diagram connecting human browsing, assisted discovery, and authorized agent actions to one business system.

The agentic Internet: from websites to an ecosystem of agents and bots

Cloudflare’s bot report is a starting point for a bigger business question: how should companies connect human discovery, AI assistants, and authorized agent actions?

The agentic Internet adds another audience to your business: software that discovers information, compares options, and requests actions for people. My view is that companies should prepare a connected ecosystem of readable content, trusted interfaces, and controlled operations while keeping their websites useful and accessible to human visitors.

This is ImadDhin’s analysis, written on November 7, 2026, prompted by Cloudflare’s report published on July 1, 2026. The reported findings below belong to Cloudflare. The architecture choices, illustrative examples, and recommendations that follow are my interpretation, not measured outcomes or predictions that every company has already reached the same stage.

Original ImadDhin diagram connecting human browsing, assisted discovery, and authorized agent actions to one business system.

What does Cloudflare’s report actually tell us?

Cloudflare reports that AI training represented 52% of crawler requests classified by purpose in June 2026, compared with 22% in spring 2025; mixed-use crawlers exceeded 36%. It also describes pressure on referral-based publishing and emerging content licensing. These are findings from Cloudflare’s July report, whose methodology cites Cloudflare Radar and its 2026 Investor Day presentation. The crawler figures are not a census of every website, nor the share of all Internet traffic attributable to autonomous agents.

I would treat that report as a reason to examine your own business, rather than as a forecast to copy into a strategy deck. Look at which information is being requested, which visitors become customers, and which workflows require a person to navigate several disconnected systems. A headline about aggregate traffic cannot answer whether your catalogue is accurate, whether an assistant can understand your service, or whether a proposed action is safe.

How does a website become part of an agent ecosystem?

A useful way to design the transition is to distinguish three paths that can coexist: a person browsing, an assistant helping someone decide, and an authorized agent completing a bounded task. This is my design model, not a claim that the web has passed through three universal stages or that human browsing is ending.

Interaction pathHow a task beginsWhat the business should provide
Human browsingA person opens a page and chooses the next stepClear information, accessible controls, reliable forms
AI-assisted discoveryA person asks an assistant to research or compareCurrent, attributable answers and structured service or product information
Authorized agent actionA person delegates a defined taskAuthenticated interfaces, scoped permissions, validation, confirmation, and recovery

For a service business, this might start with a prospective client reading a case study. Another prospect might ask an assistant to compare suppliers. Later, an authorized workflow could check appointment availability or prepare an inquiry. Each path should reach the same current facts and a coherent next step. The agent path should not create a separate, less trustworthy version of the business.

Are crawlers, bots, and agents the same thing?

They are different categories of automated activity, and I recommend designing policy around purpose and authority rather than a single label. A search crawler may gather pages for an index. Training-related collection has a different purpose. A retrieval system may fetch material to answer a question. A task agent may request an operation on behalf of a user. Abusive automation belongs in its own risk assessment.

These descriptions are practical distinctions, not promises that every visitor declares an unambiguous purpose. One operator can support several uses, and a request header is not proof of identity or permission. I would document the evidence used to classify requests, the uncertainty that remains, and what the system allows while that uncertainty is unresolved.

The decision to expose public information is separate from the decision to permit a transaction. A system can make a service description easy to retrieve while refusing unauthenticated access to customer records. It can accept an authenticated request to prepare a draft while still requiring a person to approve sending it. Those boundaries should be explicit in the architecture and visible to whoever owns the workflow.

What should companies make discoverable first?

Start with the information customers need to make a sound decision: what you offer, who it is for, how the process works, what evidence supports it, and how to contact you. My recommendation is to improve a small set of useful pages before generating more content. A concise, maintained explanation is a better foundation than a large catalogue of near-duplicate pages.

For products, choose a source of truth for names, availability, specifications, and commercial terms. For services, clarify scope, deliverables, exclusions, and the next conversation. Where a detail changes frequently, establish who updates it and how outdated copies are corrected. Do not publish confidential customer information simply because it would give an assistant more context.

Keep the human experience complete. A visitor should still be able to read the main answer, understand the evidence, and submit an inquiry without depending on an animation or signing into an account. I would test the page with keyboard navigation and with scripts unavailable, then check what a text-based client receives. Useful content should survive the loss of a decorative interface.

What infrastructure turns discovery into a trusted action?

A discoverable page tells an agent what a company does; a controlled interface defines what it may do. My architecture work starts by mapping the user, the business system, the available operation, and the decision owner. Only then do I choose the agent framework or integration approach. See the Agent Engineering practice for the architecture patterns and delivery lifecycle.

For example, a booking workflow might expose an availability query and a separate request to reserve a slot. The system should verify the account and requested operation, check current availability again when committing, and return a durable confirmation. If the network fails after a reservation is recorded, a retry should recover that confirmation instead of creating a second booking. This is an illustrative design requirement, not a claim about an existing client implementation.

MCP can be one interface in that design. Its HTTP authorization specification describes authorization for restricted servers acting on behalf of resource owners. It does not remove the need to implement the company’s own business rules. I would still validate the requested action, enforce tenant boundaries, limit privileges, and record the result in the underlying system of record.

How can a business stay open without allowing every bot?

I recommend separating public discovery policy from protected operations. Public content can have a documented crawl policy, while customer records and consequential actions require actual access controls. The Robots Exclusion Protocol, RFC 9309, explicitly distinguishes crawler rules from access authorization. A robots file should not be treated as the security boundary for private information.

For a particular company, I would map the intended audiences and decide what each can read, request, or change. Keep denied operations denied even when the request arrives through a different interface. Evaluate rate limits, identity checks, and request logging against the real task rather than relaxing every control to make an assistant succeed.

Content reuse and commercial access may also need a business decision. My engineering role is to make an agreed policy enforceable and observable, not to assume that every publicly reachable page is intended for every purpose. Where a proposed policy has contractual consequences, the company should define the terms with its appropriate decision makers before those terms become automated rules.

What should you measure when fewer journeys look like a page visit?

Measure completed business outcomes alongside traffic, while keeping each signal honest about what it proves. My suggested baseline includes successful inquiries, qualified conversations, completed tasks, escalation quality, cost per completed task, and the number of corrections a person must make. A fetched page is not proof that an assistant recommended you; an assistant referral is not a booked project.

Keep conventional website analytics and operational records connected through an appropriate measurement plan. Distinguish observed referrals from unattributed visits and automated requests. Review whether a request reached the right team, whether the records were correct, and whether a user understood what happened. Avoid filling gaps in attribution with invented certainty about which model or assistant influenced the decision.

Before expanding an agent’s authority, agree on the release criteria and examine exceptions. A workflow that handles routine cases but silently mishandles unusual ones may create more work for the company. I would run representative cases, inspect failures, test recovery, and compare the results with the existing process. Expand scope when the evidence supports it, with an owner who can stop or reverse the rollout.

Where should a company start building for the agentic Internet?

Begin with one customer journey and one bounded operation that share a clear business goal. I would choose a task with an identifiable owner, accessible evidence, and a manageable consequence if something goes wrong. Map the current journey, improve the public information, define the integration contract, and prove the operation before widening autonomy.

A practical first engagement can produce a discovery map, an access matrix, a prototype against a controlled dataset, and an evaluation plan. It should also identify what should remain a conventional form, deterministic function, or human decision. An agent is a design choice within the system, not a requirement for every step. The AI agent development page explains how that work can be scoped.

As founder and architect, my role is to connect that first use case to the company’s broader infrastructure: its product interfaces, information ownership, identity, integrations, and operating model. The goal is a useful presence across human and agent-mediated journeys, supported by software the team understands and can maintain. Let's map the architecture to map the first workflow and the boundaries around it.

Frequently asked questions

Does the agentic Internet mean websites will disappear?

That is not the assumption behind this article. I recommend supporting human visitors, assisted discovery, and authorized agent actions together, with consistent information across those interfaces.

Are all bots autonomous AI agents?

No. Crawling, retrieval, training-related collection, task execution, and abusive automation have different purposes. Do not infer permission or user delegation merely from an automated request.

Does the report say 52% of all web traffic is AI agents?

The figure concerns training-related requests within a classified crawler dataset. It does not measure delegated agent actions across the entire web.

Do we need MCP before improving our website?

No. Begin with accurate public information and a useful customer journey. Consider MCP or other integration interfaces when a defined task requires them and the access model is clear.

What is a sensible first agent project?

Choose one bounded workflow with an accountable owner, measurable outcomes, and clear permission limits. Start with observation or drafts, evaluate representative cases, and expand only when the results justify it.

Make your company ready for human and agent journeys.

A conversation about your first useful workflow.

Book an architecture call

Architecture patterns, infrastructure, and the delivery cycle.

Explore Agent Engineering

Keep reading