This could be interesting, but all the example code is so poor, and the extent of the fix so limited that it feels like this is 'AI' for the sake of it.
Stringly-typed, catch-log-ignore, generates warnings in IDE anyway...
Fixed 90 of 125 errors that were found by a different technology... probably.could have been fixed by one developer in 1-2 days anyway - how long did this take?
> all the example code is so poor, and the extent of the fix so limited
I agree about the limitations, but look from another perspective, is it good use of time asking experienced engineers who get paid 300-500k/year to go and fix these issues? Probably no.
Would it bring them a joy finding simple leaks and fixing them? Probably no again.
Then this tool is a nice addition to the engineers
That BufferedReader constructors is not supposed to throw, unless the reader is null or you're out of memory. If your out of memory you should probably abort the program execution because the difficulty of correctly recovering from an OOM exception is between hard and impossible.
Edit:
Disregard the above, it was too naive.
It cannot leak because of a stronger warranty. In the JLS, it is specified that the try-with-resources track successfull initialisation of AutoClosable resources, not successful assignation. So one try with resources can desugar into the logical equivalent of multiple nested try-catch-finally !
> difficulty of correctly recovering from an OOM exception is between hard and impossible.
In Java out of memory is signaled with an OutOfMemoryError which is a throwable (and can be caught) but is not technically an exception. Errors should generally never be caught and cannot be recovered from, which is how they differ from exceptions.
It's not great, but you can always catch and retry if your belief is that the GC will free enough memory to allow the attempt to continue after the memory pressure subsides.
Let's say you get 1/100 requests that are randomly sent to your process. That 1 takes 100x the average memory usage of the others. You could spin it out to different services to better handle the weird one-off, but that doesn't always make sense. Sometimes you just need to be ok with working the 100x job and let the other 99 get progressive falloff retry. Different solutions are always possible.
> It's not great, but you can always catch and retry if your belief is that the GC will free enough memory to allow the attempt to continue after the memory pressure subsides.
No, you cannot. Catching, for example, StackOverflowError (which inherits from Error) can lead to very strange deadlocks and such (if locking is relying on try-finally discipline, as it should), even if you do "almost nothing" before re-throwing.
It's a huge hornet's nest of weirdness to even attempt to catch anything which derives directly from Error. (Rather than RuntimeException/Exception.)
EDIT: There are some really strange subclasses of Error now that I think about it. E.g. VirtualMachineError ... I don't think I've ever seen that in any logs, thankfully, but what exactly is the program (running on the failing VM) supposed to do if that is thrown? It'd be like trying to carry on or log an error if suddenly 1==2 turned out to be true.
> There are some really strange subclasses of Error now that I think about it. E.g. VirtualMachineError
An OutOfMemoryError is a VirtualMachineError. The Java runtime doesn't technically contain the idea of "finite memory". The language sort of assumes there's an infinite amount of memory. When there isn't and the VM is forced to throw an OutOfMemoryError it's technically a breach of the abstraction of the language and the VM is unable to continue working.
It's possible to recover enough to cleanly save the user work and shutdown. You have to mesure of much memory you need for a clean shutdown and allocate a byte[sizeForShutdown] at the start of the application in the top level exception handler (the only one that should catch that error) and free that byte array before doing anything else.
Now you can argue that this is not truly recovering from exception but it is a lot better than what you can do with most of the other Error subclasses. I said most because there are easy to recover from Error like the StackOverflowError where you just have to fail the operation or request that caused the error.
Does the JLS actually guarantee that if you free that byte array then the extra memory will be immediately available? I thought there could potentially be some lag. The safer approach would be to create all of the objects you need for a clean shutdown during program launch and keep them around. Then you won't need to allocate any memory in the exception handler.
There were static code analyzers like findbugs able to do stuff like this 20 years ago. These days, your IDE should tell you off as well. And hopefully teams configure their builds to catch stuff like this using any of the available plugins for this.
LLMs seems overkill for this. But of course letting it do general code reviews is not a bad idea. And static code analyzers aren't perfect. Doing this at scale might be a bit tricky though; you'd have to go file by file. I assume Uber might have many millions of lines of code to screen. And maybe their developers aren't that concerned with addressing warnings.
But going file by file might be simple enough. And you can probably screen for important stuff with some decent system prompt and instruct it to check for potential resource leaks and similar issues and produce detailed bug reports with some priority if it finds them and a diff that addresses the issue. Could be a nice use case for agentic AIs to screen code like this.
These existing tools (spotbugs, findings, sonarqube, null away, checker framework, pmd, etc) provide detection but not resolution. Resolution thru LLM is what Uber proposes.
Resolution of the kind of resource leaks that SonarQube finds is usually the work of a few seconds. And it should be integrated into the developer workflow in such a way that those bugs are found and fixed before the code even gets checked in. In other words, if that code is even making it into a repo where GenAI can find and fix it then you have deeper engineering process problems. Fix the root cause rather than patching around them with more tools.
And once SonarQube flags a resource leak to a developer, a competent developer would learn from that and never make the same mistake again.
I think you'd be surprised, how many people do production code WITHOUT automated formatting / linting / static analysis.
Now these people and their managers are staring at the shiny new thing, the LLMs. They missing googling skills are fixed by LLMs ont heir everyday life, so they think a magic "code quality tool" (AI) would help them in they work.
To answer your question, obviously there are many linters/analyzers for Java (and for other languages too), both for static and dynamic code analysis, formatters, everything.
I'm not sure I fully understand the purpose of this tool. Modern IDEs are quite good at detecting these types of issues. And if they don't catch something, I often just copy and paste the method with some context-into an LLM and ask it to find the problem. It usually handles all the necessary changes for me.
This architecture is promising. Large legacy code bases can have static analysis violations in the 1000s which devs never have the time to address. I've seen sonar tube reports that require man years to resolve everything.
This could be interesting, but all the example code is so poor, and the extent of the fix so limited that it feels like this is 'AI' for the sake of it.
Stringly-typed, catch-log-ignore, generates warnings in IDE anyway...
Fixed 90 of 125 errors that were found by a different technology... probably.could have been fixed by one developer in 1-2 days anyway - how long did this take?
Mostly agree with these points, but it does say that it automates some of the fixes in complex cases that the non-AI tool couldn't handle.
Besides, gotta start somewhere. It's probably a PoC, for a platform that will eventually handle all sorts of things.
> Besides, gotta start somewhere. It's probably a PoC, for a platform that will eventually handle all sorts of things.
I agree and I think we should give folks leeway to make progress but this seems to be the qualifier for nearly every GenAI demo I’ve seen
> all the example code is so poor, and the extent of the fix so limited
I agree about the limitations, but look from another perspective, is it good use of time asking experienced engineers who get paid 300-500k/year to go and fix these issues? Probably no.
Would it bring them a joy finding simple leaks and fixing them? Probably no again.
Then this tool is a nice addition to the engineers
> try (BufferedReader reader = new BufferedReader (new FileReader (filePath) )){
won't FileReader leak if the BufferedReader constructor throws?
That BufferedReader constructors is not supposed to throw, unless the reader is null or you're out of memory. If your out of memory you should probably abort the program execution because the difficulty of correctly recovering from an OOM exception is between hard and impossible.
Edit: Disregard the above, it was too naive.
It cannot leak because of a stronger warranty. In the JLS, it is specified that the try-with-resources track successfull initialisation of AutoClosable resources, not successful assignation. So one try with resources can desugar into the logical equivalent of multiple nested try-catch-finally !
> difficulty of correctly recovering from an OOM exception is between hard and impossible.
In Java out of memory is signaled with an OutOfMemoryError which is a throwable (and can be caught) but is not technically an exception. Errors should generally never be caught and cannot be recovered from, which is how they differ from exceptions.
It's not great, but you can always catch and retry if your belief is that the GC will free enough memory to allow the attempt to continue after the memory pressure subsides.
Let's say you get 1/100 requests that are randomly sent to your process. That 1 takes 100x the average memory usage of the others. You could spin it out to different services to better handle the weird one-off, but that doesn't always make sense. Sometimes you just need to be ok with working the 100x job and let the other 99 get progressive falloff retry. Different solutions are always possible.
> It's not great, but you can always catch and retry if your belief is that the GC will free enough memory to allow the attempt to continue after the memory pressure subsides.
No, you cannot. Catching, for example, StackOverflowError (which inherits from Error) can lead to very strange deadlocks and such (if locking is relying on try-finally discipline, as it should), even if you do "almost nothing" before re-throwing.
It's a huge hornet's nest of weirdness to even attempt to catch anything which derives directly from Error. (Rather than RuntimeException/Exception.)
EDIT: There are some really strange subclasses of Error now that I think about it. E.g. VirtualMachineError ... I don't think I've ever seen that in any logs, thankfully, but what exactly is the program (running on the failing VM) supposed to do if that is thrown? It'd be like trying to carry on or log an error if suddenly 1==2 turned out to be true.
> There are some really strange subclasses of Error now that I think about it. E.g. VirtualMachineError
An OutOfMemoryError is a VirtualMachineError. The Java runtime doesn't technically contain the idea of "finite memory". The language sort of assumes there's an infinite amount of memory. When there isn't and the VM is forced to throw an OutOfMemoryError it's technically a breach of the abstraction of the language and the VM is unable to continue working.
It's possible to recover enough to cleanly save the user work and shutdown. You have to mesure of much memory you need for a clean shutdown and allocate a byte[sizeForShutdown] at the start of the application in the top level exception handler (the only one that should catch that error) and free that byte array before doing anything else.
Now you can argue that this is not truly recovering from exception but it is a lot better than what you can do with most of the other Error subclasses. I said most because there are easy to recover from Error like the StackOverflowError where you just have to fail the operation or request that caused the error.
Does the JLS actually guarantee that if you free that byte array then the extra memory will be immediately available? I thought there could potentially be some lag. The safer approach would be to create all of the objects you need for a clean shutdown during program launch and keep them around. Then you won't need to allocate any memory in the exception handler.
C# has Roslyn analyzers for this. Doesn't java have something similar ? Is AI appropriate technology for static analysis ?
There were static code analyzers like findbugs able to do stuff like this 20 years ago. These days, your IDE should tell you off as well. And hopefully teams configure their builds to catch stuff like this using any of the available plugins for this.
LLMs seems overkill for this. But of course letting it do general code reviews is not a bad idea. And static code analyzers aren't perfect. Doing this at scale might be a bit tricky though; you'd have to go file by file. I assume Uber might have many millions of lines of code to screen. And maybe their developers aren't that concerned with addressing warnings.
But going file by file might be simple enough. And you can probably screen for important stuff with some decent system prompt and instruct it to check for potential resource leaks and similar issues and produce detailed bug reports with some priority if it finds them and a diff that addresses the issue. Could be a nice use case for agentic AIs to screen code like this.
These existing tools (spotbugs, findings, sonarqube, null away, checker framework, pmd, etc) provide detection but not resolution. Resolution thru LLM is what Uber proposes.
Resolution of the kind of resource leaks that SonarQube finds is usually the work of a few seconds. And it should be integrated into the developer workflow in such a way that those bugs are found and fixed before the code even gets checked in. In other words, if that code is even making it into a repo where GenAI can find and fix it then you have deeper engineering process problems. Fix the root cause rather than patching around them with more tools.
And once SonarQube flags a resource leak to a developer, a competent developer would learn from that and never make the same mistake again.
I'm working in the developer productivity space.
I think you'd be surprised, how many people do production code WITHOUT automated formatting / linting / static analysis.
Now these people and their managers are staring at the shiny new thing, the LLMs. They missing googling skills are fixed by LLMs ont heir everyday life, so they think a magic "code quality tool" (AI) would help them in they work.
To answer your question, obviously there are many linters/analyzers for Java (and for other languages too), both for static and dynamic code analysis, formatters, everything.
It's not identifying the leaks. In their architecture they use Sonar qube for the detecting. The LLM provides the fix.
I'm not sure I fully understand the purpose of this tool. Modern IDEs are quite good at detecting these types of issues. And if they don't catch something, I often just copy and paste the method with some context-into an LLM and ask it to find the problem. It usually handles all the necessary changes for me.
Is this open sourced? It would be super useful if we can run against open source community projects to see the accuracy
This architecture is promising. Large legacy code bases can have static analysis violations in the 1000s which devs never have the time to address. I've seen sonar tube reports that require man years to resolve everything.