will do. It would help me a lot if there's a better one out there. Let me play around with it after earnings season. what's your view on shortcut.ai? any pathway for them to become useful? why are they so bad now?
Great article. I went down this road too. LLM’s understand financial definitions conceptually, but then in practice always make up the data and forecasts. And like you noticed it often doesn’t add up correctly. I started writing AI investment reports (ywr-intelligence.world) where I pull the financial data first to an SQL database so there is no hallucination and I’m dealing with with hard data. Then I ask the LLM to be creative on things like the ‘Bull Case’. But it’s not the holy grail you and we all want where we chat with an LLM and they build a model. I think the counterintuitive reality is that even though they are computer programs they don’t like structure and structured data. It’s why they always struggle pulling data from a table or Excel, even though for us it’s easy.
I agree. LLMs don't do well with data. It's inherent in the design. That's why there's so much inconsistency. The inconsistency makes me doubt that LLMs will ever get there - if it's always wrong, then I feel like there's a fix, but if it's inconsistent, then I feel like there's no fix...that's my current thinking on the matter, but who knows, AI is interesting and maybe down the road, they can combine LLM with another different architecture to fix the issue, but I don't think it's as easy as "just give it 1-2 years and we'll solve the issue."
I find it interesting that this new amazing technology, actually struggles with structured data, which obviously old computers excel at. And like you say, it seems like something which should be fixable... but if it really is a challenge, then that brings into question how useful it will be in a corporate setting.
YEP! I was thinking the exact same thing. There's fundamentally something off with LLM when it comes to dealing with numbers and data. And maybe LLMs are fooling us into thinking it's amazing because we use language so imprecisely. For example, I can complete that last sentence in 20 different ways, and it would all mean the same thing, but you can't do that with data.
Saw your tweet. Great note. Similar experience. I have just been playing with shortcut.ai for a few hours (I was too late for the beta test) on the $40/hour trial plan. Task was to build a 3-statement model, DCF and trading multiple based valuation sheets for a US-listed company. Shortcut could not even source historical financials for FY24. I uploaded a csv file with actual and consensus data (from Koyfin). AI just repeatedly getting lost. Balance sheets not balancing etc. Would have been quicker to do myself. Entry-level white-collar work is not yet dead!
Once you find a mistake, it almost take longer to spot check than to do it from scratch yourself. At least we don’t have that problem with shortcut cause it’s all garbage at this point hahahha
This is so detailed!
Can you write an honest review for us, TabAI, as well?
Would love feedback, we did a zero shot showdown and found ourselves to be 3x faster and better.
https://www.youtube.com/watch?v=tk6MXaPScd4
We're built by PE, IB veterans and Columbia AI Researchers.
Test us for yourself, takes 5 clicks to install, feedback appreciated :)
will do. It would help me a lot if there's a better one out there. Let me play around with it after earnings season. what's your view on shortcut.ai? any pathway for them to become useful? why are they so bad now?
Great article. I went down this road too. LLM’s understand financial definitions conceptually, but then in practice always make up the data and forecasts. And like you noticed it often doesn’t add up correctly. I started writing AI investment reports (ywr-intelligence.world) where I pull the financial data first to an SQL database so there is no hallucination and I’m dealing with with hard data. Then I ask the LLM to be creative on things like the ‘Bull Case’. But it’s not the holy grail you and we all want where we chat with an LLM and they build a model. I think the counterintuitive reality is that even though they are computer programs they don’t like structure and structured data. It’s why they always struggle pulling data from a table or Excel, even though for us it’s easy.
I agree. LLMs don't do well with data. It's inherent in the design. That's why there's so much inconsistency. The inconsistency makes me doubt that LLMs will ever get there - if it's always wrong, then I feel like there's a fix, but if it's inconsistent, then I feel like there's no fix...that's my current thinking on the matter, but who knows, AI is interesting and maybe down the road, they can combine LLM with another different architecture to fix the issue, but I don't think it's as easy as "just give it 1-2 years and we'll solve the issue."
I find it interesting that this new amazing technology, actually struggles with structured data, which obviously old computers excel at. And like you say, it seems like something which should be fixable... but if it really is a challenge, then that brings into question how useful it will be in a corporate setting.
YEP! I was thinking the exact same thing. There's fundamentally something off with LLM when it comes to dealing with numbers and data. And maybe LLMs are fooling us into thinking it's amazing because we use language so imprecisely. For example, I can complete that last sentence in 20 different ways, and it would all mean the same thing, but you can't do that with data.
Saw your tweet. Great note. Similar experience. I have just been playing with shortcut.ai for a few hours (I was too late for the beta test) on the $40/hour trial plan. Task was to build a 3-statement model, DCF and trading multiple based valuation sheets for a US-listed company. Shortcut could not even source historical financials for FY24. I uploaded a csv file with actual and consensus data (from Koyfin). AI just repeatedly getting lost. Balance sheets not balancing etc. Would have been quicker to do myself. Entry-level white-collar work is not yet dead!
Once you find a mistake, it almost take longer to spot check than to do it from scratch yourself. At least we don’t have that problem with shortcut cause it’s all garbage at this point hahahha
I think it’s because our cofounder spent years in PE, IB, etc. 10 hours a day, 7 days a week.
She dogfoods our product daily.
All these drive our model choice, prompt, tool engineering, context decision, etc.
Also some RL secret sauce by our team who has papers published at neurips, iclr, icml ;)