Rhoai Llamastack

RHOAI 3.3上的LlamaStack

使用LlamaStackDistribution自定义资源在OpenShift AI 3.3上部署具有MCP服务器集成的LlamaSstack服务器。

建筑

External vLLM Endpoints
  (Inference + Embedding)
         |
         v
+---------------------+         +-------------------------+
| LlamaStack Server   |--SSE--->| OpenShift MCP Server    |
| (rh-dev, port 8321) |         | (local, port 8080)      |
|                     |         |   --cluster-provider     |
| Providers:          |         |     in-cluster           |
|  - vllm-inference   |         +-------------------------+
|  - vllm-embedding   |                  |
|  - milvus (inline)  |             in-cluster SA
|  - rag-runtime      |                  |
|  - model-context-   |                  v
|    protocol         |         Local OpenShift API Server
|                     |
| Tool Groups:        |         +-------------------------+
|  - builtin::rag     |--SSE--->| OpenShift MCP Server    |
|  - builtin::websearch         | (remote, port 8080)     |
|  - mcp::openshift   |         |   --kubeconfig          |
|  - mcp::openshift-  |         |     /etc/mcp/kubeconfig  |
|    remote (optional) |         +-------------------------+
+---------------------+                  |
         |                          kubeconfig +
         v                          SA token
+---------------------+                  |
| PostgreSQL          |                  v
| (KV + SQL storage)  |         Remote OpenShift API Server
+---------------------+

先决条件

• 启用LlamaStack操作员的OpenShift AI 3.3

(llamastackoperator.managementState: Managed 在您的DataSciencesCluster CR中）

• oc CLI已登录到集群
• envsubst 可用（部分 gettext --预装在大多数系统上）
• 外部vLLM推理端点URL（以结尾 /v1)和API令牌
• 外部vLLM嵌入端点URL和API令牌

文件

快速开始

# 1. Clone the repo
git clone https://github.com/ravisharma5/rhoai-llamastack.git
cd rhoai-llamastack

# 2. Configure your environment
cp .env.example .env
# Edit .env with your vLLM endpoints and API tokens

# 3. Login to your OpenShift cluster
oc login --server=https://api.your-cluster.com:6443

# 4. Deploy everything
./deploy.sh

# 5. Verify
oc get pods
LLAMA_URL=https://$(oc get route llama-stack-server -o jsonpath='{.spec.host}')
curl -sk $LLAMA_URL/v1/health

手动部署

如果您更喜欢逐步控制部署：

1.创建命名空间

oc new-project llama-stack

2.配置和创建机密

cp .env.example .env
# Edit .env with your values, then:
source .env
envsubst "
MCP_SERVER_URL = "http://openshift-mcp-server:8080/sse"

response = requests.post(
    f"{LLAMA_STACK_URL}/v1/responses",
    json={
        "model": "your-inference-model",
        "input": "List all pods in the llama-stack namespace",
        "tools": [
            {
                "type": "mcp",
                "server_label": "openshift",
                "server_url": MCP_SERVER_URL,
                "require_approval": "never",
            }
        ],
        "stream": False,
    },
    verify=False,
    timeout=120,
).json()

for item in response["output"]:
    if item["type"] == "mcp_call":
        print(f"Tool: {item['name']}({item.get('arguments', '{}')})")
    elif item["type"] == "mcp_call_output":
        print(f"Output: {item['output'][:500]}")
    elif item["type"] == "message":
        content = item.get("content", "")
        if isinstance(content, list):
            for c in content:
                if c.get("type") == "output_text":
                    print(f"Assistant: {c['text']}")

直接工具调用

curl -sk -X POST $LLAMA_URL/v1/tool-runtime/invoke \
  -H "Content-Type: application/json" \
  -d '{
    "tool_name": "pods_list_in_namespace",
    "kwargs": {"namespace": "llama-stack"},
    "tool_group_id": "mcp::openshift"
  }'

重要说明

ConfigMap密钥必须为 config.yaml

操作员在以下位置装载ConfigMap /etc/llama-stack/ 和设置 LLAMA_STACK_CONFIG=/etc/llama-stack/config.yaml.钥匙 必须 是 config.yaml:

# Correct
oc create configmap llama-stack-run-config --from-file=config.yaml=llamastack-config.yaml

# Wrong -- causes "Could not resolve config" error
oc create configmap llama-stack-run-config --from-file=run.yaml=llamastack-config.yaml

使用 base_url 不 url 在vLLM提供程序配置中

RHOAI 0.4.x vLLM提供程序使用 base_url.使用 url 被默默忽略：

# Correct
config:
  base_url: ${env.VLLM_URL}

# Wrong
config:
  url: ${env.VLLM_URL}

使用 userConfig 不 userConfigMapRef

# Correct
userConfig:
  configMapName: llama-stack-run-config

# Wrong -- silently ignored
userConfigMapRef:
  name: llama-stack-run-config

PostgreSQL镜像使用 POSTGRESQL_* 环境变量

Red Hat PostgreSQL映像预期 POSTGRESQL_USER, POSTGRESQL_PASSWORD, POSTGRESQL_DATABASE --不是 POSTGRES_*.

默认情况下，NetworkPolicy会阻止外部路由流量

操作员创建了一个限制进入的网络策略。CR包括 network.allowedFrom.namespaces: ["*"] 允许来自OpenShift的流量 路由器。

MCP服务器配置

安全模式

已启用的工具集

添加远程OpenShift群集

默认 openshift-mcp-server 使用集群内的ServiceAccount与本地集群通信。要访问不同的OpenShift集群，请部署第二个MCP服务器pod，并使用指向它的kubeconfig。

1.在远程集群上创建ServiceAccount

登录到远程群集并创建只读ServiceAccount：

# Log in to the remote cluster
oc login https://api.remote-cluster.example.com:6443

# Create namespace and ServiceAccount
oc new-project mcp
oc create sa mcp-viewer -n mcp

# Grant cluster-wide read access
oc adm policy add-cluster-role-to-user cluster-reader \
  system:serviceaccount:mcp:mcp-viewer

2.生成令牌并构建kubeconfig

# Generate a time-bound token (adjust duration as needed)
TOKEN="$(oc -n mcp create token mcp-viewer --duration=8h)"
API_SERVER="$(oc whoami --show-server)"

# Build a dedicated kubeconfig file
oc login --server="$API_SERVER" --token="$TOKEN" \
  --kubeconfig="$HOME/.kube/mcp-remote.kubeconfig"

# Verify
oc --kubeconfig="$HOME/.kube/mcp-remote.kubeconfig" get nodes

注： ServiceAccount令牌在指定的持续时间后过期。你将 需要重新生成令牌并在密钥过期时更新密钥。使用时间更长 开发持续时间(--duration=168h 7天）。

3.在LlamaStack集群上使用kubeconfig创建一个Secret

切换回LlamaStack集群并将kubeconfig存储为Secret：

# Log in to the LlamaStack cluster
oc login https://api.llamastack-cluster.example.com:6443
oc project llama-stack

# Create the Secret from the kubeconfig file
oc create secret generic remote-cluster-kubeconfig \
  --from-file=kubeconfig=$HOME/.kube/mcp-remote.kubeconfig

4.部署远程MCP服务器

创建 openshift-mcp-server-remote.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openshift-mcp-server-remote
  labels:
    app: openshift-mcp-server-remote
    app.kubernetes.io/component: mcp-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: openshift-mcp-server-remote
  template:
    metadata:
      labels:
        app: openshift-mcp-server-remote
        app.kubernetes.io/component: mcp-server
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: mcp-server
          image: quay.io/containers/kubernetes_mcp_server:latest
          args:
            - "--port"
            - "8080"
            - "--disable-destructive"
            - "--toolsets"
            - "core,config,helm"
            - "--kubeconfig"
            - "/etc/mcp/kubeconfig"
            - "--disable-multi-cluster"
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: kubeconfig
              mountPath: /etc/mcp
              readOnly: true
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            capabilities:
              drop:
                - ALL
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 10
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
            initialDelaySeconds: 3
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 100m
              memory: 128Mi
      volumes:
        - name: kubeconfig
          secret:
            secretName: remote-cluster-kubeconfig
---
apiVersion: v1
kind: Service
metadata:
  name: openshift-mcp-server-remote
  labels:
    app: openshift-mcp-server-remote
spec:
  selector:
    app: openshift-mcp-server-remote
  ports:
    - name: http
      port: 8080
      targetPort: http
  type: ClusterIP

部署它：

oc apply -f openshift-mcp-server-remote.yaml
oc wait --for=condition=available deployment/openshift-mcp-server-remote --timeout=120s

5.在LlamaStack配置中注册工具组

将新工具组添加到 llamastack-config.yaml 在...之下 tool_groups:

tool_groups:
  # ... existing toolgroups ...
  - toolgroup_id: mcp::openshift-remote
    provider_id: model-context-protocol
    mcp_endpoint:
      uri: http://openshift-mcp-server-remote:8080/sse

更新配置映射：

oc create configmap llama-stack-run-config \
  --from-file=config.yaml=llamastack-config.yaml \
  --dry-run=client -o yaml | oc apply -f -

重新启动LlamaStack以获取新的工具组：

oc rollout restart deployment/llama-stack-server

6.查询远程集群

将响应API与 openshift-remote 服务器标签：

response = requests.post(
    f"{LLAMA_STACK_URL}/v1/responses",
    json={
        "model": "your-inference-model",
        "input": "List all pods in the default namespace",
        "tools": [
            {
                "type": "mcp",
                "server_label": "openshift-remote",
                "server_url": "http://openshift-mcp-server-remote:8080/sse",
                "require_approval": "never",
            }
        ],
        "stream": False,
    },
    verify=False,
    timeout=120,
).json()

这 server_label 控制哪个集群收到请求： "openshift" 击中本地集群， "openshift-remote" 点击遥控器。要添加更多集群，请使用不同的名称重复步骤1-5。

代币更新

当SA令牌过期时，重新生成它并更新Secret：

# On the remote cluster
oc login https://api.remote-cluster.example.com:6443
TOKEN="$(oc -n mcp create token mcp-viewer --duration=8h)"
API_SERVER="$(oc whoami --show-server)"
oc login --server="$API_SERVER" --token="$TOKEN" \
  --kubeconfig="$HOME/.kube/mcp-remote.kubeconfig"

# On the LlamaStack cluster
oc login https://api.llamastack-cluster.example.com:6443
oc project llama-stack
oc create secret generic remote-cluster-kubeconfig \
  --from-file=kubeconfig=$HOME/.kube/mcp-remote.kubeconfig \
  --dry-run=client -o yaml | oc apply -f -

# Restart the MCP server to pick up the new token
oc rollout restart deployment/openshift-mcp-server-remote

更新配置

oc create configmap llama-stack-run-config \
  --from-file=config.yaml=llamastack-config.yaml \
  --dry-run=client -o yaml | oc apply -f -

拆除

./teardown.sh
# Or manually:
oc delete llamastackdistribution llama-stack-server
oc delete route llama-stack-server
oc delete configmap llama-stack-run-config
oc delete -f openshift-mcp-server.yaml
oc delete -f postgres.yaml
envsubst < llama-stack-secret.yaml | oc delete -f -

Claude代码技能

此repo包含LlamaStack技能，用于 克劳德代码 在 .claude/skills/llamastack/SKILL.md。它为Claude提供了有关LlamaStack API、部署模式、MCP集成和常见陷阱的上下文，因此可以帮助您使用此项目。

它涵盖了什么

• LlamaStack v0.4.x API参考（响应API、工具调用、聊天完成）
• MCP服务器注册和身份验证模式（配置级别与请求时间）
• 使用LlamaStackDistribution CR进行OpenShift部署
• 配置文件引用(config.yaml 使用env-var插值）
• 9个可以节省调试时间的陷阱（无声字段名不匹配、缺少路由等）

设置

当您在此项目目录中使用Claude Code时，该技能会自动加载。无需额外设置——Claude代码读取 .claude/skills/ 从repo根目录。

您可以直接调用它：

/llamastack deploy    # deployment help
/llamastack api       # API reference
/llamastack mcp       # MCP server integration
/llamastack gotchas   # common pitfalls

或者只是自然地问问题，克劳德会在相关的时候退出技能。

需求

• 克劳德代码 已安装CLI
• 跑 claude 从该仓库的根目录

故障排除

CrashLoopBackOff中的LlamaStack吊舱：

• 检查日志： oc logs -l app.kubernetes.io/instance=llama-stack-server
• 验证ConfigMap密钥是否 config.yaml
• 验证 base_url （不是 url)在vLLM提供程序配置中

路线返回503：

• 检查网络策略： oc get networkpolicy -o yaml
• 确保 network.allowedFrom.namespaces: ["*"] 在CR中设置

MCP工具未出现：

• 验证 mcp::openshift 在...里 llamastack-config.yaml 在...之下 tool_groups
• 验证 model-context-protocol 提供者在 providers.tool_runtime
• 检查LlamaStack日志中的MCP连接错误