@lobehub/chat/packages/prompts/promptfoo/knowledge-qa/eval.yaml

Lobe Chat - an open-source, high-performance chatbot framework that supports speech synthesis, multimodal, and extensible Function Call plugin system. Supports one-click free deployment of your private ChatGPT/LLM web application.

description: Test knowledge base Q&A with context retrieval and answer generation

providers:
  - openai:chat:gpt-5-mini
  - openai:chat:claude-3-5-haiku-latest
  - openai:chat:gemini-flash-latest
  - openai:chat:deepseek-chat

prompts:
  - file://promptfoo/knowledge-qa/prompt.ts

tests:
  - vars:
      context: "React is a JavaScript library for building user interfaces. It was developed by Facebook and is now maintained by Facebook and the community. React uses a virtual DOM to efficiently update and render components. Key features include component-based architecture, JSX syntax, and state management through hooks."
      query: "What is React and who developed it?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should accurately explain what React is and mention Facebook as the developer"
      - type: contains-any
        value: ["React", "JavaScript", "library", "Facebook"]
      - type: not-contains
        value: "I don't know"

  - vars:
      context: "TypeScript is a strongly typed programming language that builds on JavaScript by adding static type definitions. It was developed by Microsoft. TypeScript code compiles to clean, simple JavaScript code which runs on any browser, Node.js environment, or any JavaScript engine that supports ECMAScript 3 or newer."
      query: "How does TypeScript relate to JavaScript?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should explain the relationship between TypeScript and JavaScript, mentioning type definitions"
      - type: contains-any
        value: ["TypeScript", "JavaScript", "type", "compiles", "strongly typed"]
      - type: not-contains
        value: "不知道"

  - vars:
      context: "Node.js是一个基于Chrome V8引擎的JavaScript运行时环境。它使用了事件驱动、非阻塞I/O模型，使其轻量而高效。Node.js的包管理器npm是世界上最大的开源库生态系统。"
      query: "Node.js有什么特点？"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should describe Node.js features in Chinese, mentioning event-driven and non-blocking I/O"
      - type: contains-any
        value: ["Node.js", "事件驱动", "非阻塞", "JavaScript", "运行时"]
      - type: not-contains
        value: "I don't know"

  - vars:
      context: "Docker is a containerization platform that allows developers to package applications and their dependencies into lightweight, portable containers. Containers share the OS kernel but run in isolated user spaces. This makes applications more consistent across different environments."
      query: "How can I deploy my app with Docker?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should provide helpful information about deploying with Docker based on the containerization context. It should mention containers, packaging, or provide deployment-related guidance."
      - type: contains-any
        value: ["Docker", "container", "package", "deploy"]

  - vars:
      context: "GraphQL is a query language for APIs and a runtime for fulfilling those queries with existing data. Unlike REST APIs that require multiple requests to different endpoints, GraphQL allows clients to request exactly the data they need in a single request."
      query: "What are the benefits of using GraphQL over REST?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should compare GraphQL and REST, highlighting GraphQL's advantages such as single request capability or requesting specific data"
      - type: contains-any
        value: ["GraphQL", "REST"]

  - vars:
      context: "Machine learning algorithms can be categorized into supervised, unsupervised, and reinforcement learning. Supervised learning uses labeled data to train models, unsupervised learning finds patterns in unlabeled data, and reinforcement learning learns through trial and error with rewards."
      query: "Can you explain blockchain technology?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should indicate that the provided context is about machine learning, not blockchain, and cannot answer the blockchain question based on the context"
      - type: contains-any
        value: ["machine learning", "cannot", "不能", "no information", "context", "does not contain"]

  - vars:
      context: ""
      query: "How do I set up a web server?"
    assert:
      - type: llm-rubric
        provider: openai:gpt-5-mini
        value: "The response should provide helpful information about setting up a web server using general knowledge"
      - type: contains-any
        value: ["server", "web", "setup", "install", "configure"]
      - type: not-contains
        value: "cannot answer"