The Cloud Should Adapt to Its Applications, Not the Other Way Around!

Sachin Ashok†*, Karuna Grewal§*, Talha Waheed†, Gangmuk Lim†, Gerard Matthew†¶, Aditya S Prerepa†,
P. Brighten Godfrey†, Justin Hsu§, Radhika Mittal†
†University of Illinois Urbana-Champaign   §Cornell University   Amazon   *Equal contribution    HotNets ’26
Paper 2-minute video Website by Sachin Ashok
  1. Status quoProduction controllers rely on CPU and latency, and treat each app as an opaque box
  2. ProblemKnowing more about the app, its insights, helps. But research extractors are brittle, so production doesn’t rely on them
  3. Why nowToday, LLM agents can reason across code, configs and telemetry, and combine tool outputs
  4. SolutionAn Application Insight Plane that extracts each insight once and shares it
  5. Early resultsAble to extract the insight (concurrency model) in a general manner from 100 services, and up to 30% lower p95 on leveraging the insight
Read the full argument
Status quo
Production controllers, such as autoscalers and load balancers, treat each application as an opaque box. They see only basic signals: request rate, latency and CPU.
Opportunity
Controllers could manage applications much better if they knew application insights, such as call graphs and concurrency models, and tuned their approach to each application.
Problem
Research prototypes use such insights, but each builds its own extractor. These extractors are tied to one system and break as the application changes, so few of these ideas reach production.
Why now
LLM agents can read code and telemetry, work across frameworks, and run existing tools.
Solution
An architectural change: an Application Insight Plane (AIP) between applications and controllers, where LLM agents
  1. extract each insight once,
  2. keep it valid as the application changes, and
  3. share it with every controller.
Early results
Across 116 services in 8 apps, AIP’s concurrency model provider extracted a model for each service with source code (100 of them) and returned no model for ones without (16 of them). With AIP’s concurrency insight, Envoy cuts p95 latency by up to 30%.
Open questions
Which insights to standardize, how to keep them valid, and how to share them safely.

This page is interactive. Follow the orange instructions in each section.

1. Status quo: production controllers see each application as an opaque box

However, a much richer, fine-grained application-level view is possible.

Switch to the application-level view, then pick an insight.

Insight:

Application

an opaque box to its controllers

Basic signals

    Research systems already use insights like these. So why don’t production controllers? Section 2 →

    2. Problem: each system reinvents the same wheel

    Research systems already use insights. But each one builds its own extractor, even when they need the same insight. Here, every controller needs the same wheel: the application’s call graph.

    Change the application, or add a new controller. Then switch on AIP and try again.

    Real systems that build their own extractors

    Each extractor is tied to a particular language, runtime or instrumentation, is not kept up to date as the application changes, and serves only one controller.

    This is part of the reason why research systems rarely reach production.

    LLM agents can now do this extraction across languages and frameworks. Section 3 →

    3. Our solution: an Application Insight Plane

    LLM agents can now read code, configs and telemetry across languages and frameworks, and run existing tools. AIP uses them to extract each insight once, and shares it with every controller.

    Click through the four steps to follow one insight.

      One insight in the registry can serve many kinds of controllers. Section 4 →

      4. One plane serves many kinds of controllers

      Click a “Use case” to see how a controller uses that insight. Dashed boxes mark pairs that do not apply.

      Applications change all the time. How does an insight stay correct? Section 5 →

      5. Keeping an insight valid as the code changes

      The provider extracts again only when a change touches what the insight was built from.

      • Click “Fetch insight”.
      • Make a change, then click “Fetch insight” again.
      • Click a change again to undo it.

      Make a change

      
    
      
    
      
      
      Concurrency model provider
      no insight yet
      
      
        
      
      
      
    
        
  
        
  
        Please note: the provider’s output on this page is precomputed.
        
  
        Does the insight help a real controller, and does extraction work beyond one app? Section 6 →
        

        


        
  
        6. Early results
        
  
        
    
        Generality
        
      
        • 8 microservice applications from 7 benchmark suites, 116 services in total.
          • 
        
        • 100 services had source code. The provider extracted a concurrency model for each one.
          • 
        
        • 16 services had only deployment files. The provider abstained instead of guessing.
        
      
        LLM:
        GPT-5.5 via OpenAI Codex, the same setup for every application
        Schema:
        one for every service, ConcurrencyModel.v1
        Audit:
        a second agent checks each result against the code and configuration lines it cites
        
      
        Concurrency mechanisms found: 11 kinds across 105 components
        
      
        Servlet thread pool (Tomcat)
        53
        Goroutine per RPC (Go gRPC)
        12
        Thread per connection (C++ Thrift)
        11
        Event loop (Node.js, Netty)
        11
        Thread pool (Python gRPC, Java executors)
        5
        Queue worker (RabbitMQ consumers)
        4
        Goroutine per request (Go HTTP)
        3
        Runtime-managed pool (Java gRPC, ASP.NET Core)
        2
        Process pool (Gunicorn, uWSGI)
        2
        Async runtime (Go)
        1
        PHP worker pool (Apache)
        1
        
    
        
    
        Utility
        
      
        • Up to 30% lower p95 latency under overload.
        • Goodput at most 0.4% lower.
        
      
        Switch the setup to see which pool sets the limit.
        
      
        
      
      
        
    
        
  
        

        



        
  
        7. Open questions
        
  
        
    
        Which insights to standardize?
        The few kinds many controllers need, each with one schema.
        
    
        How to keep them valid?
        Record what each insight was derived from, and check it again when that changes.
        
    
        How to share them safely?
        Give each controller only what it needs, such as a thread limit rather than the code.
        
  
        

        


        
  
        8. The paper in two minutes