Costello Research SEC/PCAOB

GenAI brings us closer to automating investment expertise

Jennifer Anzaldi — Thu, 22 Aug 2024 18:39:34 +0000

GenAI brings us closer to automating investment expertise Jennifer Anzaldi Thu, 08/22/2024 - 14:39

In This Story

Yi Cao

Long Chen

Large language models (LLMs) such as ChatGPT and Google Gemini excel at being trained on large data-sets to generate informative responses to prompts. Yi Cao, an assistant professor of accounting at the Donald G. Costello College of Business at �鶹��Ƶ, and Long Chen, associate professor and area chair of accounting at Costello, are actively exploring how individual investors can use LLMs to glean market insights from the dizzying array of available data about companies.

Long Chen and Yi Cao

Their new working paper, co-authored with Jennifer Wu Tucker of the University of Florida and Chi Wan of University of Massachusetts Boston, examines AI’s ability to identify “peer firms,” or product market competitors in an industry.

Cao explains the significance of selecting peers by relating this process to the real-estate market. “The capital market is similar to the real-estate market in that a firm’s value is partially determined by the value of its peers. In the real-estate market, we price a home based on the value of comparable properties in the neighborhood, or the so-called 'comps.' In our paper, we aim to leverage the power of LLMs to identify comps for evaluating firm value.”

This task is at least as difficult as it is essential. It takes much time, skill and effort to gather, aggregate and manage data to select peers. However, the researchers reasoned that LLMs could do a lot of the heavy lifting of data aggregation and analysis for the individual investors, and produce a list of peers comparable in validity to that identified by human experts.

“The advantage is in the capability to utilize all the information potentially out there so that it is at least performing as well as other traditional methods that can help us investors and researchers,” says Cao.

For the study, Chen and Cao employed Bard from Google, now known as “Gemini,” as their LLM of choice because “Bard has a greater ability to utilize its pre-training data, which is arguably larger than ChatGPT’s and with more parameters,” says Cao.

After defining “product market competition” and forming a prompt for Bard, the researchers instructed Bard to limit its knowledge pool to a specific year within the period 1981-2023, in order to avoid “look-ahead bias,” i.e., future information scrambling the results.

“We need to understand that LLMs are actually a very powerful, new tool, unmatched in their efficiency, ability to process vast amounts of information at a low cost, and accessibility to the general public.”

They limited focal firms to large, publicly listed companies as there is less data out there for smaller or private firms. In all, the data-set comprised more than 300,000 focal firm-years.

On average, the LLM could generate about seven peer firms for a focal firm, a number that is similar to the SEC recommendations on how firms should disclose their segments.

The researchers then compared the LLM’s performance to the lists generated by three human experts for a set of 40 leading computer software companies. The average overlap was a little over 40 percent, greater than expected.

They also compared the AI-identified peer lists to two alternative systems for identifying peers: the federal government’s Standard Industrial Classification (SIC) codes and Text-based Network Industry Classification (TNIC), which compares firms based on linguistic similarities in their 10-K filings. The LLM’s output overlapped significantly with TNIC’s. Plus, the peers identified by the LLM were generally a better fit than those from SIC and TNIC, as their monthly stock returns hewed closer to the focal firm.

But TNIC outperformed the LLM in identifying peers for mid-sized firms within the sample, indicating that it is not a clear-cut case of universal LLM superiority.

“It’s especially beneficial for individual investors—as all the cost concerns that we’re talking about are especially relevant for them,” Chen adds.

Regarding the future of LLM, Chen states, “There are always costs and benefits associated with using generative AI. It is uncertain whether current systems will soon be obsolete.” When asked about the SEC adopting an AI tool for investors, Chen emphasizes that users need to understand the pros and cons of using AI to make their informed judgments “because AI cannot be held responsible for the information it provides or for how it is utilized.”

Chen concludes, “We need to embrace this new technology, but we must recognize that it is not yet in a perfect state. Competition to improve the technology is fierce. Our findings might just represent the lower bound of the effectiveness of the technology.”

Topics

A.I. & Innovation - Costello

Costello Research Innovation Strategy

Costello Research Private Funds

Costello Research Market Efficiency

Artificial Intelligence

Do former regulators improve the quality of audits?

Marianne Klinker — Mon, 08 Jan 2024 13:44:01 +0000

Do former regulators improve the quality of audits? Marianne Klinker Mon, 01/08/2024 - 08:44

A �鶹��Ƶ professor unpacks the complex, nuanced impact of the “revolving door” between industry and regulators in the accounting world.

In their auditing capacity, accounting firms, such as the “Big Four”—Deloitte, EY, KPMG and PWC—function as watchdogs for publicly traded companies. They’re tasked with ensuring financial disclosures are accurate and above board. But who watches the watchdogs?

The Sarbanes-Oxley Act of 2002 created the Public Company Accounting Oversight Board (PCAOB), whose mandate is to monitor and inspect firms conducting audits of public companies and report its findings to the public. Yet, in a pattern familiar from other government-industry configurations, the border between regulator and regulated is often less a brick wall than a revolving door. Large accounting firms, including the Big Four, have been hiring more and more PCAOB employees, especially since 2010, when the board expanded its remit to include internal control audits.

Steve Maex

“Firms began moving in the direction of hiring significant numbers of PCAOB employees after a string of weak inspection reports in the early 2010s, ostensibly to get a sense of what they could do to better comply with auditing standards,” says Steve Maex, an assistant professor of accounting in the Donald G. Costello College of Business at �鶹��Ƶ.

Maex’s recently published paper in Review of Accounting Studies, co-authored with Jagan Krishnan and Jayanthi Krishnan from Temple University, evaluates the effects of such hiring. The researchers tracked the audit quality of large accounting firms over the period in which this hiring emerged and explored the relationship between the hiring and a variety of audit outcomes.

“There’s no single proxy that can be used to capture the many dimensions of audit quality,” Maex points out. So, the team used various indicators, including clients’ financial restatements, discretionary accruals (disparities between reported income and cash flow), the accuracy of internal control opinions issued by the auditor, and audit fees.

The researchers found that clients of firms that hired ex-PCAOB employees issued fewer restatements in general, which suggests fewer egregious errors on the part of the auditor. Further, for clients that would be more pre-disposed to misstating their financials, this quality-enhancing effect was also identified in terms of lower discretionary accruals, higher accuracy of internal control audit opinions, and higher fees that clients were willing to pay. In other words, the former PCAOB personnel seemed to help their firms focus on the highest risk issues and clients in their portfolios.

The researchers found that clients of firms that hired ex-PCAOB employees issued fewer restatements in general, which suggests fewer egregious errors on the part of the auditor.

Maex and his coauthors surmise that these professionals, during their experience with PCAOB, acquired “regulatory audit quality expertise” that would not be as easily accessible to those without such experience. “The practitioners we interviewed said that [at the PCAOB] you get exposed to a variety of different audit practices and processes and develop an understanding of what works and does not work well. This contrasts with many audit firm partners who start and finish their career in the same accounting firm and therefore may only be exposed to their own firm’s methodology and a handful of clients that they support,” he explains.

This broad-based, generalizable skillset involves a greater aptitude for gauging and addressing risk. “To the extent that they can help the firm identify high-risk clients, that right off the bat can help them allocate resources intelligently. These individuals are going to be really good at finding strategies, solutions, and methodologies to handle the firm’s high-risk clients more effectively.”

For Maex, these findings form part of an active stream of accounting research studying the role of regulatory oversight on auditing firms. Furthermore, the work is relevant in light of recent events in which ex-PCAOB auditors at KPMG tipped off the firm about the board’s inspection plans. The ensuing scandal sparked debate about possible misconduct enabled by the revolving door.

Maex’s research highlights that the movement of former regulators to accounting firms offers potential upsides for audit quality notwithstanding these ethical considerations. Rather than seeking to inhibit transfers on ethical grounds, regulators could explore different types of talent exchanges with firmer guardrails against misconduct. The SEC’s Professional Accounting Fellows program, which admits experienced accounting professionals for a limited period of time and under fixed parameters, could serve as a model.

“The goal of the regulators, we like to think, is the same as the audit firms: to ensure audit quality is as strong as it could be,” Maex says. “If there were no conversations about how to achieve that between the two, that could be problematic. The challenge is balancing that against the negative outcomes, which receive significant publicity when they occur.”

Topics

Costello Research Internal Audit

Costello Research Recruiting

Costello Research Internal Audit

More Costello College of Business Faculty Research

Are there upsides to “overboarding”?
July 14, 2025
“Doing well by doing good”? There’s a framework for that
July 2, 2025
In the workplace, relationships equal reality
May 28, 2025
Why it doesn’t—and shouldn’t—always pay to be a super-successful CEO
May 7, 2025
Study: Left-handed CEOs are more innovative
April 29, 2025

In This Story

Steve Maex

�鶹��Ƶ researcher helps the SEC walk the talk

Marianne Klinker — Tue, 31 Jan 2023 14:37:09 +0000

�鶹��Ƶ researcher helps the SEC walk the talk Marianne Klinker Tue, 01/31/2023 - 09:37

Research by �鶹��Ƶ Accounting Professor Bret Johnson, a former SEC staff accountant and academic fellow, shows how seemingly mundane intra-agency policies can have unintended effects that benefit Wall Street over Main Street.

The U.S. Securities and Exchange Commission (SEC) has long seen itself as a friend to the retail investor, doing everything it can to ensure that financial markets offer both large and small players roughly equal opportunity to succeed. Most recently, that mandate provided the rationale for a slate of sweeping rule changes that would, among other things, shine a light on the opaque business arrangements between wholesale brokers and trading apps such as Robinhood. It adds up to what some industry insiders are calling the largest-scale SEC intervention in nearly 20 years.

Bret Johnson

To its credit, the SEC is just as committed to promoting fairness and transparency in-house as it is to ambitious industry reforms. The agency even invites accounting academics to research its inner workings via the SEC Academic Fellowship Program. �鶹��Ƶ School of Business accounting professor and former SEC staff accountant Bret Johnson was selected for the year-long program starting in August 2020.

Johnson says, "My role was to be a liaison between the SEC and the academic community…Often, my research uncovers some problem or something they can improve on, some inefficiencies. And I found they are very open to embracing this type of research."

For example, Johnson's recent research paper in Review of Accounting Studies co-authored by Michael Iselin of the University of Minnesota, Jacob Ott of the London School of Economics, and Jacob Raleigh of Monash University, finds that while the SEC's monitoring arm – the Division of Corporation Finance (DCF), where he worked for six years – seems to focus less on firms with a high percentage of retail ownership, the Division of Enforcement (DOE) appears to take a harder line with these firms. In other words, irregularities are less likely to be spotted early and addressed through non-punitive means when the firm in question has a larger proportion of retail investors.

"My role was to be a liaison between the SEC and the academic community…Often, my research uncovers some problem or something they can improve on, some inefficiencies. And I found they are very open to embracing this type of research." - Bret Johnson

The SEC's enforcement-heavy approach may disadvantage retail investors because once the enforcement action becomes common knowledge, the "Main Street" investors who hold disproportionate stock in the target company will lose out when the share price falls. In this way, the SEC does not seem to be acting in accordance with its stated mission to protect mom and pop investors.

Johnson can only guess at what's going on here. He speculates that firms that attract retail investors may share other characteristics that cause the SEC to flag them for enforcement rather than monitoring. For instance, the paper finds that retail ownership was positively associated with firm visibility and performance volatility, and negatively associated with external monitoring and structural complexity. The discrepancy may also have to do with how resources are distributed between the DOE and DCF.

In other cases, the agency's good-faith attempts to increase market transparency on behalf of less savvy investors can backfire. Based on analytics from the publicly available filings database on the SEC website, Johnson's 2020 paper in The Accounting Review, co-authored by Michael S. Drake and Jacob R. Thornock of Brigham Young University, and Darren T. Roulstone of The Ohio State University, concluded that institutional users (e.g. investment banks, hedge funds, asset managers, and financial institutions) were profiting handsomely from their search activities on the online resource, while retail investors saw little to no benefit. Abnormal search volume from Wall Street was associated with future returns for the focal firms of up to 10.5 percent. No such pattern was seen on Main Street.

In addition, how the SEC balances the need for transparency against its relationships with companies can have unintentional knock-on effects. Johnson’s 2022 paper in Management Science, co-authored by Marshall A. Geiger and Abdullah Kumas of the University of Richmond, and Keith L. Jones of the University of Kansas, found that for companies that had just received an SEC comment letter, there was a sizeable uptick in sell-offs among mutual funds and other major players during the short period (initially 45 days, then shortened to 20 days) before the letter went public. The best explanation was that high-ranking executives were sharing non-public information with investors close to the company concerned.

The 20-day privacy window is meant to prevent companies’ confidential information from being accidentally released through an over-hasty process. But it may also create information asymmetry that benefits industry insiders at the expense of retail investors – the very favoritism that the agency intends to combat.

There are no simple answers to the questions posed by Johnson’s research. Even seemingly mundane intra-agency decisions, such as resource allocations and the length of the confidentiality window for comment letters, can have complicated ripple effects. Sometimes, it takes an informed outsider to trace these ripple effects and draw evidence-based conclusions. "I am happy to see that the SEC continues to solicit – and take seriously – academic research into its policies and practices," Johnson says. After all, sweeping industry reforms may make headlines, but fine-tuning how the agency itself functions is a quieter but no less material way of promoting fairness and transparency in financial markets.

Topics

Costello Research Non-Financial Disclosure

Impact Fall 2023

More School of Business Faculty Research

Are there upsides to “overboarding”?
July 14, 2025
“Doing well by doing good”? There’s a framework for that
July 2, 2025
In the workplace, relationships equal reality
May 28, 2025
Why it doesn’t—and shouldn’t—always pay to be a super-successful CEO
May 7, 2025
Study: Left-handed CEOs are more innovative
April 29, 2025

In This Story

Bret Johnson

How Investors Profit from 'Private' SEC Correspondence

Jennifer Anzaldi — Wed, 27 Apr 2022 16:10:43 +0000

How Investors Profit from 'Private' SEC Correspondence Jennifer Anzaldi Wed, 04/27/2022 - 12:10

In This Story

Bret Johnson

Until 2004, the U.S. Securities and Exchange Commission (SEC) “comment letters”—queries sent to companies regarding accounting disclosures—were withheld from the general public. As a result, a cottage industry sprang up around selling material obtained in response to Freedom of Information Act requests. To stem the flood of paperwork and ensure more equal access to information, the SEC announced comment letters would be uploaded to a database, called EDGAR, accessible to the public on the SEC’s website.

However, the SEC allowed a 45-day pre-release window (later shortened to 20 days) after the close of each review, so that managers’ confidential treatment requests and SEC internal documentation would not be inadvertently released. This exclusivity window created a brief interval of information asymmetry, in which information intended for—and potentially of great interest to—investors is available only to corporate insiders.

Bret Johnson, assistant professor of accounting at �鶹��Ƶ, recently co-authored a paper in Management Science finding that SEC comment letters are leaking out among investors close to the company concerned, who then use it to their advantage. Technically, such information-sharing violates Regulation Fair Disclosure (FD), which prohibits companies from sharing secrets with network partners such as institutional investors and analysts.

“I think one of the hardest regulations to enforce is Regulation FD, because it’s so hard to tie down whether information was privileged when they received it,” Johnson says. “There is so much we can’t observe as researchers, let alone the SEC.”

Nonetheless, Johnson and his co-authors[1] found evidence of abnormal trading activity in the time between issuance and public release of comment letters. “Mutual funds in our study, on average, net sold $724,000 [worth of] equity securities per firm in the six days following the receipt of each SEC comment letter,” the paper states. Additionally, the amount of abnormal selling per letter increased as the exchanges with the SEC grew to two or more letters—supplying yet more evidence of a causal relationship, since deeper and thornier issues, i.e. those most meaningful to investors, presumably require more rounds of correspondence. The selling activity was also higher for comment letters that addressed more critical issues.

The researchers’ intuition that insiders were sharing the contents of comment letters with close associates seemed to be confirmed by a higher intensity of selling activity around firms whose directors were more well-connected, and those with an above-median percentage of dedicated institutional ownership.

Going further, the researchers determined that these sell-offs generated significantly beneficial financial outcomes for the mutual funds in the 45 days after a private comment letter was sent. The more abnormal selling mutual funds engaged in during the private period, the greater the monetary rewards.

To be sure, a comment letter from the SEC isn’t necessarily a red flag that would draw investor attention. It is estimated that only about half of 10,000 reviews result in a comment letter being issued, and many types of comments are quite routine. Letters could be triggered by anything from a simple filing error to shifting SEC priorities that go beyond any one company. However, they can also be leading indicators of impending trouble.

Johnson says that comment letters in general contain “signals [that] could be very subtle or small pieces of information. It might not be a big impairment or write-down. It may just be that the investors are learning a little bit more about management, how transparent they’re willing to be with the SEC staff reviewers…They’re learning these more qualitative aspects, as well as what the numbers are saying.”

Johnson is intimately familiar with how the SEC operates, having worked for six years as a staff accountant in the agency’s Division of Corporation Finance, which handles the filing review process and the issuance of comment letters. In addition, he held a one-year academic fellowship at the SEC ending in July 2021, where he served as a “liaison between SEC staff and the academic community, keeping them abreast of developments related to their policies.”

As with the information leakage paper, Johnson’s research often zeros in on issues that have policy implications for the SEC, and it has been warmly received by his former colleagues: “They are hungry to receive academic research that could help improve their policies and practices,” Johnson says.

He hopes that his research will prompt a debate within the agency about the optimal length of time that should transpire before comment letters are added to EDGAR. Further shortening of the 20-day window, or even releasing letters in real time, could be considered, but these options would possibly undermine the agency’s goal of fostering frank and open dialogue with companies. Johnson recalls SEC staff accountants saying to him, in effect, “When we issue a comment to a company, we want them to really defend or articulate their accounting positions. We encourage them to push back.” It’s unclear how curtailing the window of confidentiality would affect managers’ willingness to engage.

Johnson says his findings should also inspire managers to get their interactions with the SEC right the first time, thus avoiding drawn-out exchanges that are likely to be overheard by insider-adjacent investors. “An audit partner I was recently talking to about this process said, ‘You don’t want to get into a pen-pal relationship with the SEC.’”

[1] Marshall A. Geiger and Abdullah Kumas of University of Richmond, and Keith L. Jones of University of Kansas.

Topics