Oakleaf’s AI & Analytical Solutions: Using AI for Loan File Compliance and Bias Reviews

Oakleaf AI Solutions

Oakleaf is enhancing the mortgage industry with its AI and analytical solutions, designed to assist mortgage lenders in efficiently compiling, summarizing, and organizing underwriting notes, including defects or exceptions and appraisal summaries. With Oakleaf’s solutions, lenders can identify trends such as lending or appraisal bias, improve quality control review processes, and effectively manage compliance reviews.

Empowering Capital Markets with Insightful Analytics

In addition to supporting mortgage lenders, Oakleaf can provide capital markets participants with crucial insights into the relationship between underwriting quality/decisions and performance. This is of particular significance in the non-qualified mortgage (NQM) market, which is composed of various lenders with unique guidelines and exceptions that make risk assessment challenging. Through the integration of Large Language Models (LLMs) and statistical modeling, Oakleaf aims to clarify the potential links between loan underwriting exceptions and performance. The technology holds promise for diverse sectors, notably the emerging Scratch and Dent RMBS market, where it could significantly enhance the understanding of the intricacies of underwriting and loan programs and their potential impact on performance.

Balancing Sound Lending with AI

Oakleaf’s AI solutions can help mortgage lenders strike the right balance between sound lending decisions and expanding credit to underserved borrowers. By analyzing underwriting decisions and loan attributes, Oakleaf can facilitate identifying patterns in approvals and denials, and can also help in detecting bias in appraisals, servicing notes or other written reports.

Innovative Loan Defect Identification Using LLMs and Logistical Regression Analysis

Oakleaf’s most recent project involved using GPT to analyze defects identified during forensic re-underwriting reviews of approximately 3,800 loans, covering a total of 5,700 defects. The project then utilized a logistical regression to determine whether specific loan attributes, or combinations thereof, could effectively identify loans with a higher likelihood of defects.

Step 1: Using AI to Classify Underwriting Exceptions

Oakleaf has a long history of conducting forensic re-underwriting reviews to identify representation and warranty breaches and mortgage file defects. As part of these reviews, Oakleaf underwriters identify relevant defects for each loan and draft narratives detailing these defects. Typically, loans are selected for re-underwriting based on losses incurred. Recognizing that a more targeted selection process could save clients both time and money, Oakleaf developed a methodology that combines LLM analysis with logistical regression analysis. This innovative approach analyzes historical forensic re-underwriting results to pinpoint loans most likely to be defective, specifically by finding relationships among defects and certain loan attributes such as amortization type, credit score, documentation type, etc., or combinations thereof.

Defect Category Classification by GPT

In a detailed analysis of loan defects, Oakleaf’s AI team utilized GPT to categorize defects into seven categories, accurately reflecting the nature of each defect with an 80-90% accuracy rate. The categories included debt-to-income ratio (DTI), loan underwriting, loan-to-value (LTV), and occupancy misrepresentation, among others. Below is a breakdown of the findings:

Defect Category

By carefully crafting GPT prompts to include detailed background information and summarization instructions for each defect narrative, Oakleaf’s AI team ensured the accurate categorization of defects.

Following this categorization, Oakleaf utilized its statistical modeling capabilities, combining GPT’s results with logistical regression analysis to identify whether a relationship exists between loan attributes and underwriting defects. Oakleaf was able to tie certain loan attributes, such as interest-only or negative amortization (neg am) features, to specific defect categories.

Broadening the Scope of Data Analysis

While this project originally focused on forensic re-underwriting, the possibilities for this type of data analysis in the arenas of quality control and predictive modeling are extensive. By using specific loan underwriting exceptions or defects as explanatory variables, this innovative technology opens new avenues for advancing loan quality assessment and prediction.

Step 2: Enhancing Quality Assurance with AI

In our effort to enhance quality assurance (QA) processes, we have turned to AI, incorporating the advanced GPT-4 model, to conduct thorough reviews of our findings. Traditional QA methods, while effective, are time-consuming and susceptible to human error. AI offers a promising alternative, capable of analyzing large amounts of text data with precision and speed.

The effectiveness of AI in our QA process lies in the instructions, known as prompts, provided to it. These prompts are crucial for guiding the AI’s analysis and ensuring it focuses on the relevant aspects of our data. By tailoring the AI to adopt different perspectives and thinking styles, we’ve significantly improved the quality of our reviews.

We initially programmed GPT to categorize defects based on underwriters’ notes, acting as a creative problem solver. We then introduced a modification that allows the AI to also perform as a quality control reviewer, enabling a layer of critical analysis that was previously unattainable.

This innovative approach has yielded meaningful results. In a review of 5,700 breach narratives, the QA agent identified 5% of loans requiring further review, as it did not agree with the classification assigned in the initial exercise described in Step1 above. GPT’s nuanced reasoning, when given the proper instructions, not only streamlines QA reviews but also improves the selection process by identifying loans where contradictory responses are generated.

Step 3: Utilizing Logistic Regression to Identify Potentially Defective Loans

Oakleaf ran a logistic regression on GPT’s results to identify loan attributes with a higher likelihood of underwriting defects. Our focus was on fully documented loans, especially since non-fully documented loan programs, such as those stating income, showed a high correlation with DTI-related defects.

The results showed that certain attributes, such as interest-only or neg am loans, were more strongly related to defects. However, it also indicated that original loan attributes offer limited predictive value (about 65% accuracy) regarding loan underwriting quality. This limitation likely stems from the dataset’s constraints, which lacked comprehensive attributes like employment type or detailed underwriting program information.

Our project demonstrates Oakleaf’s capability in building various complex predictive models. This analysis showed limitations of traditional metrics in capturing loan quality, emphasizing the need for a deeper, more nuanced understanding of underwriting factors. This experience has enriched our analytical toolkit and enhanced our team’s ability to adapt and refine our methodologies in response to unexpected findings.

To learn more about Oakleaf’s AI product offerings and consultative services, please contact Suzanne Mistretta at Suzanne.mistretta@oakleaf.com.

Sign Up for Newsletter Updates

Oakleaf at a Glance

The Oakleaf Group is a Washington DC-based, management consulting firm focused exclusively on the financial services, banking, and mortgage industry. Our team consists of industry subject matter experts, risk management professionals, and technologists who work together to help solve our clients’ toughest problems. Oakleaf strives to enhance the financial and regulatory industry with data-driven, model-informed, analytics-enabled, and operationally rigorous services and solutions.

See Who We Are  |  Meet Our Leadership Team

Join The Oakleaf Team

Oakleaf is the trusted advisor and services provider to many of the some of of the most notable mortgage businesses, federal agencies, and financial institutions in the world. We look for exceptional people from a variety of fields and professions to help solve our clients most challenging issues.

Join Oakleaf and put your talents and skills to work with our leading financial, banking, and mortgage client organizations.

See The Work We Do

As a mortgage and financial services consulting firm, we assist companies throughout the mortgage and financial services industry, including banks and non-bank mortgage firms, GSEs, regulators, government agencies, law firms, insurance companies, asset managers, hedge funds, and other financial clients. We work closely with management to identify and prioritize challenges, and we apply deep industry expertise that drives performance, resulting in performance transformation across the organization.

See how we support our clients and their teams in tackling their most complex matters. Or contact us if you want to discuss anything further.