AI Chatbot Commentary

Introduction

AI chatbots are starting to show up across consumer-facing platforms, including fintech companies. Usage is rapidly increasing, and LLMs are increasingly fielding personal finance questions. TD Bank’s most recent AI Insights Report found the share of Americans using AI to help manage their personal finances jumped from 10% to 55% in a single year. A NerdWallet study found 26% of Americans had tried AI to help with personal finances. Curious as to the capabilities and limitations of these new features, we jumped in and tested them.

Four are covered here: ChatGPT’s finance module, SoFi’s Coach, Robinhood’s Cortex, and Public’s AI Research Assistant and Generated Assets. Each was built for different purposes and had different limitations and strengths. I asked each model some basic allocation questions, budgeting questions, and investment research questions. ChatGPT’s finance module was unsurprisingly the least constrained in what it would do.

A note: We are reporting on our own experience with a limited number of inquiries into the chatbots covered. This is not intended to be a comprehensive benchmarking of the capabilities of each of these products.

ChatGPT Finance

ChatGPT’s finance module connects your financial accounts through what is called an ‘aggregator’. An aggregator allows a secure connection to a financial account to then pull in balance and transaction information. ChatGPT’s financial module did not have a narrow intended use the way the others on fintech platforms did. It is an LLM attached to an aggregator, which means it will attempt to answer nearly anything you put to it, from a monthly budget to a company research report.

That range is appealing, and it is also where some concerns arose. The module was ready to provide insights before it had collected an appropriate amount of context. It suggested reconsidering a stock-to-bond ratio before it knew an age or time horizon for the funds. A human financial planner will ask questions and build a profile of someone before starting the planning exercise. ChatGPT was eager to answer regardless of how much it knew about me. When asked if it knew my age, it noted that it inferred I had a long time horizon from my portfolio positioning, but ‘should not guess my age from that’.  This is backward and dangerous logic. It was inferring my investing time horizon from my positioning instead of first gathering some additional information so it could guide me appropriately.

It was also ready to be persuaded. At first, it flagged my investments as too domestically focused, with only 10% exposure to international equities. Still, when asked whether the domestic concentration had helped performance over the past decade, it quickly moderated its call to increase international equities significantly.

It flagged a large cash balance and recommended to ‘give the cash a specific job’, not connecting that I had already defined a large portion of that cash as earmarked for a specific purpose, or that in a previous session I recommended holding this amount or more in cash as an emergency reserve.

The same gap showed up in planning. Asked to build a financial plan, it did not recognize that a rental property belongs in future income projections or asset balances.

ChatGPT Finance can do a great deal. A single dashboard for all of your finances is valuable, and it can certainly help provide generic planning guidance. But the user has to define the parameters correctly, be ready to question its assumptions, know what a financial planner would need to build a good plan, and watch for inaccuracies and misread data. The aggregator integration is the substantial feature here; accounts come in and stay current without manual entry. Beyond basic insights, this is a tool that needs a savvy user directing it for more complex needs and analysis.

SoFi Coach

Coach is there to help you use the SoFi platform, which is built on the premise that lending, banking, and investing can happen in one place. SoFi also can aggregate information from held-away accounts.  Coach was ready to build a budget, analyze accounts, and work through financial planning questions.

It handled that work well on a college savings plan (which is one goal with a known horizon) where it provided some useful guidance on how to save for education. When it came to equity vs fixed income allocations and international holdings, its approach was more measured and consistent. When asked for an opinion on equity and fixed income, it first paused to confirm my age or time horizon and gather more context. After highlighting my domestic stocks focus, it was not as ready to be persuaded by domestic outperformance over the past decade. It instead gave some insights on the benefits of diversification.

Overall, through the budgeting, asset allocation, and light planning questions, Coach seemed to have more defined parameters and was ready to ask follow-ups or gather more context when needed.

It is not a stock researcher and does not present itself as one. Asked about public companies, it offered to help set up a framework for analyzing them rather than pulling financial data and producing reports. When asked to run a screener for companies, I got a similar answer about setting up a framework but was not willing to create a list of individual names.

Robinhood Cortex

Cortex is built as a light researcher and educator: summaries of account holdings and what is driving returns, trade-offs between different allocation decisions, and education on investing topics.

Of the products, it seemed the most cautious in its responses.  Its ability to screen stocks was constrained to a handful of criteria.  It screens on sector and industry, company size, price and performance, 52-week positioning, volume and liquidity, valuation, dividends, analyst sentiment, and options activity.

Within its available criteria, every category describes a company’s state; none describes change over time. A request for companies growing earnings more than 15% per year was outside of its functionality. Similarly, a request for a research report on an individual company was primarily a snapshot in time of the company.

When asked about equity and bond allocations, Cortex was careful not to provide personalized insights.  It was similarly generic when asked about international vs domestic allocation. It did help put together a planning framework for an education savings goal and some light budget analysis.

Public

Public’s AI features aim higher on the research and portfolio construction front.

The research assistant can turn an idea into either a screen or an individual company. The information it provided seemed reliable, and it was capable enough to screen companies, generate investment ideas, and produce a note on an individual name worth building on. It was ready to screen based on historical earnings growth as well as a wide variety of other thematic or technical criteria.  When asked to provide a research report, it provided information about growth, a bear and a bull case, and it was an informative report.

The research assistant was clear that it was there for research and not planning. It declined to answer budgeting questions and pointed me in the direction of a financial advisor or planner when asked to help with my education savings goal.

Generated Assets is a separate and novel AI feature from Public. It builds a custom thematic portfolio or index from plain-language input, leveraging AI research to assemble the constituents, shows how the basket would have performed on a historical backtest, and then makes it investable. The concept recalls Motif, which shut down in 2020, and M1’s Pies, which are still running. Those platforms allowed users to build baskets of stocks and trade them as a group. Users can easily create a portfolio around themes like robotics, space travel, or insurers benefiting from AI-improved underwriting and then trade that portfolio in a coordinated way.

The Question Hanging over the Category

Public’s research assistant produced good results. And still, the question does not go away: why use it rather than going directly to ChatGPT, Claude, or Gemini? The frontier models are less constrained and frequently more thorough. With an in-platform AI, the parameters and context can be better designed for the use case, the data can be constrained to reliable sources, and context can be built in a thoughtful way for the use case.  For example, SoFi’s Coach hesitating to gather more information before answering was a nice improvement over ChatGPT’s readiness to answer. But it will be difficult for in-platform AI to compete with leading models that have fewer constraints.  This conundrum is not isolated to finance platforms. It hangs over nearly every consumer AI feature built into an existing application.

Liability and Regulatory Questions Hover

The assistants built into regulated platforms were consistently clear that they were not providing financial advice. ChatGPT Finance told us it was not providing advice only when we asked whether it was.

That is not a design quirk. SEC-regulated firms have every reason to be cautious about the line between education and regulated advice, and their products show it. What has not been answered is what happens when a model crosses that line and the advice turns out to be bad.

The stakes are not hypothetical. A NerdWallet survey found 26% of Americans have used an AI chatbot for personal finance questions. Among those who acted on the advice, 39% said it helped their financial situation, and 29% said it hurt it.

Where this is Going

This is the beginning of what will inevitably be a wave of fintech products designed to help people with their personal finances. ChatGPT Finance felt like it was winging some of the advice it gave. Future products will be trained specifically for this work, and we expect the planning to get materially more reliable.

These tools are already useful: staying organized, analyzing a budget, keeping every account visible in one place, and working through basic planning. Research agents can carry a different risk, in that they make it easy to over-complicate an investment approach that did not need complicating. And unrefined models can lead people down poor planning paths while sounding entirely sure of themselves.

A complex, multi-goal plan is a different matter. It needs someone at the controls with enough knowledge to notice when the model has drifted, missed a blind spot, or built on an assumption nobody checked. That is not a small ask of a user, and it is the reason these products are not a replacement for a financial planner.

Will Robots Replace the Advisor?

Agents will upend plenty of industries. But we do not think AI agents will replace financial advisors. Robo advisors were supposed to disrupt the industry. They instead attracted a different cohort of investors and took little market share from traditional advisors. Advisors are already leveraging AI to serve clients better. At the end of the day, financial planning and advice is a business built on trust and relationships. AI is powerful, and most Americans are not ready to trust it with their largest decisions.

What is here today is just the beginning of what will be available to users to help them with their personal finances.  While an AI tool can track your accounts in one place, provide budget insights, answer basic planning questions,, and help with research, it cannot yet replace an advisor. What is not here is the part that requires knowing which question to ask next, the real-world experience to instill trust, and a real relationship. For now, that job still belongs to the advisors and planners they hire.