Routers and Frameworks

Routers and Abstraction Layers

• Routers - Remote process that routes requests to the appropriate LLM based on the input

  • Requires the use of a single router api
    • Separate API for each LLM not needed
  • Example: OpenRouter
    • Includes 25+ free models ( has limits on number of requests per day)
    • Code Sample

• Abstraction Layers - Part of the local codebase that abstracts the complexity of the LLM calls

  • Also known as frameworks
  • Examples:
    • LangChain
    • LiteLLM
  • Called with a consistent api
  • Calls the LLM separately with the appropriate parameters

Frameworks (Abstraction Layers)

LangChain

• A framework for developing GenAI apps
  • Uses LangSmith for LLMOps (Debugging, Evaluation, Monitoring etc.)
  • Uses LangServe to create Chains as Rest APIs for depolyment