Confidently delivered: AI as analyst

Popular LLMs and AI tools can analyze your marketing in seconds. But are the results something you should trust?

by Jim Dudley, Director of Strategy

Let’s start with two facts: in a very short time, AI-powered platforms have become remarkably powerful, and they’re here to stay. AI is helping achieve breakthroughs in medicine almost weekly. In the marketing world, major Large Language Models (LLMs) like Anthropic’s Claude, OpenAI’s ChatGPT, Google’s Gemini and others dominate the news, but there is a nearly endless flow of AI-powered tools being introduced across virtually every industry that exists.

This ubiquity has led many to use these platforms for all kinds of tasks that would have been impossible to even imagine just a short while ago. In fact, our team at Anstey Hodge uses AI-powered tools daily in a range of tasks. It’s just part of our workflow at this point.

The question is: how trustworthy is the resulting analysis? Or maybe a better question is: how should you use AI analysis to best benefit your organization? To get to an answer, we asked ChatGPT to analyze our own agency website (AnsteyHodge.com) as well as a few other smaller sites owned by members of our team. Let’s break down the results into two key feedback types: technical review, and content & design review.

Technical review can be very useful (or not)

While very little in the creation of a website is a simple binary issue (good/bad), the closest we can get to that is the actual website code and technical items where best practices are very well developed. Overall, the technical feedback we were given fell into three categories: accurate and useful, accurate but not useful, and then inaccurate or misleading feedback.

Accurate and useful feedback is clearly valuable. These tended to be discrete items like missing page meta-descriptions, oversized site images (possibly slowing page load), etc. At Anstey Hodge, when we launch a new site, we test for these types of items and fix them before launch. But a site that’s been in use for a while with multiple users adding content is likely to have a few of these types of issues. Thankfully, these are usually quick-fix items, and we recommend doing so as soon as you can. 

Accurate but not very useful feedback included items where an issue was accurately detected, but either there was no negative impact to the issue, or the value of the “fix” was less than the time/effort/cost to fix it. In other words, the juice wasn’t worth the squeeze. These include things like missing H1s on system-generated pages such as blog feed pages (typically there is no title or H1 after the first page). Missing alt tags on images that are site design elements was another example. Alt tags for normal page images are absolutely required for accessibility, but a gradient pattern image that’s just part of the site background should not have an alt tag.  

Inaccurate or misleading technical feedback is the area that is most concerning. One LLM said one of our sites failed WCAG 2.1 accessibility standards, while the tool we use to test accessibility passed the website with flying colors (we ensure all sites we build meet or exceed accessibility standards). One report said we needed to remove unused JavaScript and unused images when there was neither on the site. 

The main lesson here is that while AI technical analysis will generate very confident and very detailed feedback, it takes a human or a team of people with expertise in website work to know the difference between which items in the feedback are accurate and need to be addressed now, which are lower priority and can be addressed later, and which are just wrong and should be ignored altogether.

Content & design: far more challenging analysis

In simpler areas of content and design review, such as looking for grammar issues, finding duplicate content, or checking base design items like text contrast against a background, AI tools can be very useful because they can review the entire site and not get tired or bored like a human reviewer might.

But the analysis delivered went far beyond functional design and content analysis, and that’s where we began to see some really interesting but concerning problems.

For example, part of the feedback on our own Anstey Hodge website was that the site should be much more conversion-oriented throughout. But the results included the following very specific example to illustrate the “problem”:

The homepage appears image-heavy, and the “We believe” section is values-driven rather than conversion-driven. 

For context, here is the homepage section this feedback is addressing:

The AI analyst correctly identified that these “we believe” statements are values driven. That’s the point of the statements: to share our values. Saying that this values-driven content should be changed to more conversion-driven content shows a fundamental misunderstanding of the role these statements play on a micro level, and the strategy driving our website and marketing on a macro level.

The feedback showed a fundamental misunderstanding of the strategy driving our website.

Beyond just this one example, the analysis very clearly and consistently pushed for more aggressive conversion-oriented features and language across the website: more forms, more gated content, more language about filling community occupancy, less narrative about partnership and values, etc. Design feedback was generally positive but stated that the site was image-heavy (correct), but recommended that we reduce imagery to focus more on hard statistics (such as achieving X% improvements in occupancy or SEO ranking, etc.)   

Here’s the issue with all of the feedback above: The “we believe” statements, the website design, our messaging structure, our use of images—it’s all part of a very intentional and coordinated messaging strategy that drives all of the content not just on our website but across agency marketing campaigns, new business presentations, trade show marketing, and more.

Ultimately, we observed two fundamental issues with AI analysis of these higher-level marketing concepts.

1. Context. The AI analyst lacks the context necessary to effectively review marketing at a strategic level. The AI platform really only knows what it sees on your website or in the materials you ask it to review. It doesn’t understand your brand at a human level. It doesn’t know your overall business strategy or competitive situation. It doesn’t know your brand’s history or the nuances of your target audiences and what they might expect from you.
2. Confidence. Despite this lack of contextual awareness, the AI agent didn’t just generate feedback we “might” consider, it generated a very specific list of things we should keep, things we should change, things we should add, and things we should cut. The feedback was crafted as if it was coming from an expert. The changes were very specific. The confidence and specificity of the changes listed is remarkable given that they did not align with our agency’s very carefully crafted positioning and marketing strategy. For some, the confident nature of the feedback can make it feel more reliable when it isn’t.

Treat AI analysis as a tool, not expert direction

So is there a role these AI platforms can play in the review of your website or other marketing efforts? Absolutely—as long as you keep in mind what these platforms can and can’t do. Take the feedback as a list of areas to review rather than specific action items to implement without further consideration.

If the AI analyst points out technical errors, have your team review them to see if it’s accurate, and if so, determine which changes should be prioritized based on the impact and the work required. If the AI analyst gives feedback on writing and design, ask yourself: Is the feedback aligned with my overall brand strategy and goals? If the feedback says your website or document design is too colorful, too busy, or too image-heavy, do you agree? If you’re not a design expert, ask a design expert, whether that’s on your in-house team or at your agency partner. Some of the changes suggested may improve your website. Some may undermine it. But ultimately, you and your marketing partners are the only entities with the complete picture that gives you the vision to delineate which changes might be worth pursuing.