Open Log AI

🔍 OpenSearch集中式日志服务器

🚀 具有人工智能分析功能的生产就绪集中式测井平台

*通过模型上下文协议（MCP）集成，利用实时安全监控、高级事件关联和智能自动化从Linux和Windows系统收集、处理和分析日志。*

🚀 快速开始 • ✨ 特性 • 🔧 配置 • 📊 仪表盘 • 🛡️ 安全 • 🤝 贡献

______________________________________________________________________

✨ 特性

🔍 多平台日志收集

• 🐧 Linux系统：Syslog、身份验证日志、应用程序日志、systemd日志
• 🪟 Windows事件：事件日志、安全事件、通过Winlogbeat的应用程序日志
• 🌐 网络设备：防火墙、路由器、交换机和安全设备
• ☁️ 云服务：AWS CloudTrail、Azure日志、谷歌云日志

📊 高级分析引擎

• 🔍 实时搜索：数十亿日志的亚秒级查询性能
• 📈 时间序列分析：趋势检测和异常识别
• 🔗 事件关联：跨系统事件关系映射
• 🎯 自定义仪表板：基于角色的可视化和报告

🤖 人工智能驱动的智能

• 🧠 MCP服务器：用于人工智能辅助集成的HTTP API
• 📱 流媒体分析：高效处理海量数据集
• 🔮 预测分析：基于机器学习的威胁预测
• 💬 自然语言查询：通过REST API进行类似SQL的查询

🛡️ 企业安全

• ⚡ 实时威胁检测：MITRE ATT&CK框架集成
• 🚨 自动警报：Slack/团队/电子邮件通知
• 🔐 访问控制：基于角色的身份验证和授权
• 📋 合规报告：SOC 2、PCI-DSS、HIPAA就绪

______________________________________________________________________

🏗️ 建筑

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Linux Logs    │    │  Windows Events │    │  Network Logs   │
│   (Syslog/TCP)  │    │  (Winlogbeat)   │    │   (UDP/TCP)     │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌─────────────▼───────────────┐
                    │        Logstash             │
                    │   (Processing Pipeline)     │
                    └─────────────┬───────────────┘
                                  │
                    ┌─────────────▼───────────────┐
                    │      OpenSearch Cluster     │
                    │    (Node1 + Node2 + HA)    │
                    └─────────────┬───────────────┘
                                  │
          ┌───────────────────────┼───────────────────────┐
          │                       │                       │
┌─────────▼──────────┐  ┌─────────▼──────────┐  ┌─────────▼──────────┐
│   MCP HTTP Server  │  │ OpenSearch Dashboard│ │   Direct API       │
│  (AI Integration)  │  │   (Visualizations)  │ │    (REST/SQL)      │
└──────────|─────────┘  └────────────────────┘  └────────────────────┘
           | 
           |
┌─────────▼──────────┐
│   Claud Desktop    │  
│  (AI Integration)  │  
└────────────────────┘

______________________________________________________________________

🚀 快速开始

先决条件

• 码头工人 和 Docker Compose 安装
• 4GB+内存 推荐以获得最佳性能
• 可用端口: 5601, 8000, 9200, 9600, 514, 5044

1.克隆和配置

git clone https://github.com/cybernazmul/OpenLogAI.git
cd OpenLogAi
cp .env_example .env
# Edit .env with your settings

2.启动堆栈

# Start all services
docker-compose up -d

# Check service health
docker-compose ps
docker-compose logs -f

3.克劳德桌面集成

{
  "mcpServers": {
    "opensearch-logs": {
      "command": "npx",
      "args": [
        "mcp-remote",
        "http://YOUR_SERVER_IP:8000/mcp"
      ]
    }
  }
}

4.访问服务

5.测试日志收集

# Test syslog ingestion
logger "Test message from $(hostname)"

# Verify in OpenSearch
curl -k -u admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD} \
  "https://localhost:9200/logstash-logs-*/_search?pretty&size=1"

______________________________________________________________________

🔧 配置

环境变量

所有配置都通过环境变量进行管理。复制 .env_example 向 .env 并自定义：

# OpenSearch Cluster
OPENSEARCH_INITIAL_ADMIN_PASSWORD=your_secure_password
OS_URL=https://opensearch-node1:9200
OS_USERNAME=admin
OS_PASSWORD=your_secure_password

# MCP Server Settings
MCP_HOST=0.0.0.0
MCP_PORT=8000
MCP_PATH=/mcp

# Optional: API Key authentication
OS_API_KEY=your_api_key_here

日志源配置

📧 Linux系统

# Add to /etc/rsyslog.conf or /etc/rsyslog.d/50-logstash.conf
*.* @@YOUR_SERVER_IP:514

# For specific facilities
auth.* @@YOUR_SERVER_IP:514
kern.* @@YOUR_SERVER_IP:514
mail.* @@YOUR_SERVER_IP:514

# Restart rsyslog
sudo systemctl restart rsyslog

🪟 Windows系统

# Install and configure Winlogbeat
# Download from: https://www.elastic.co/downloads/beats/winlogbeat

# winlogbeat.yml configuration:
winlogbeat.event_logs:
  - name: Application
    level: error, warning
  - name: Security
    event_id: 4624, 4625, 4648, 4656, 4719, 4720
  - name: System
  - name: Microsoft-Windows-Sysmon/Operational

output.logstash:
  hosts: ["YOUR_SERVER_IP:5044"]

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded

🐳 Docker容器

# Add to your docker-compose.yml services
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
    labels: "service,environment"

# Or use Filebeat for Docker logs

🌐 网络设备

# Cisco/Juniper/Fortigate syslog forwarding
logging YOUR_SERVER_IP:514

# pfSense/OPNsense
Configure System > Advanced > Logging
Remote Logging Options: YOUR_SERVER_IP:514

______________________________________________________________________

📊 仪表板和可视化

预构建仪表板类别

• 🔒 安全运营中心（SOC）：登录失败、权限升级、可疑活动
• 💻 系统健康：CPU、内存、磁盘使用率、服务状态
• 🌐 网络分析：流量模式、连接监控、防火墙事件
• 📱 应用程序日志：错误率、性能指标、用户活动
• 📋 合规：审计跟踪、访问日志、监管报告

示例查询

OpenSearch REST API

# Search recent security events
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/logstash-logs-*/_search" \
  -H 'Content-Type: application/json' \
  -d '{
    "query": {
      "bool": {
        "must": [
          {"range": {"@timestamp": {"gte": "now-1h"}}},
          {"match": {"syslog_program": "sshd"}}
        ]
      }
    },
    "size": 100
  }'

通过MCP进行SQL查询

-- Find failed SSH attempts
SELECT timestamp, source_ip, syslog_message 
FROM "logstash-logs-*" 
WHERE syslog_message LIKE '%Failed password%' 
AND timestamp > DATE_SUB(NOW(), INTERVAL 1 HOUR)
ORDER BY timestamp DESC;

-- Windows login analysis
SELECT user_name, logon_type, source_ip, COUNT(*) as login_count
FROM "windows-events-*"
WHERE event_id = 4624
  AND timestamp > DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY user_name, logon_type, source_ip
ORDER BY login_count DESC;

-- Top error producers
SELECT syslog_server, syslog_program, COUNT(*) as error_count
FROM "logstash-logs-*" 
WHERE syslog_severity  DATE_SUB(NOW(), INTERVAL 1 HOUR)
GROUP BY syslog_server, syslog_program
ORDER BY error_count DESC
LIMIT 20;

______________________________________________________________________

🛡️ 安全功能

🎯 自动威胁检测

• 暴力检测：>5分钟内登录失败5次
• 横向移动：不寻常的跨系统访问模式
• 权限提升: sudo, su，管理员组更改
• 数据渗漏：大额出站转账或异常访问模式
• 恶意软件指示器：流程执行异常、文件修改

📊 安全监控规则

关键警报

# Failed Authentication Attempts
- Query: syslog_message:"Failed password" OR event_id:4625
- Threshold: >5 occurrences in 5 minutes
- Action: Immediate alert + IP blocking

# Administrative Account Usage
- Query: user_name:"admin" OR user_name:"root" 
- Time: Outside business hours (6PM-8AM)
- Action: Security team notification

# New Process Execution
- Query: event_id:4688 AND process_name:NOT_IN_WHITELIST
- Scope: Critical servers only
- Action: Investigation workflow trigger

警告警报

# Unusual Login Locations
- Geographic IP analysis for user accounts
- First-time login locations
- VPN vs direct connections

# File System Changes
- Sensitive directory access (/etc, /var/log, C:\Windows\System32)
- Configuration file modifications
- Log file tampering attempts

🔐 访问控制

# Create role-based users
curl -k -u admin:${PASSWORD} -X POST \
  "https://localhost:9200/_plugins/_security/api/internalusers/analyst" \
  -H 'Content-Type: application/json' \
  -d '{
    "password": "analyst_password",
    "roles": ["logs_reader", "dashboard_user"]
  }'

______________________________________________________________________

📈 性能和扩展

资源需求

• Response: Auto-block + SOC alert

Privilege Escalation

• Pattern: sudo/su usage, admin group additions
• Scope: Non-service accounts
• Response: Immediate investigation trigger

Suspicious Process Execution

• Pattern: PowerShell/cmd.exe with encoded commands
• Pattern: Unusual parent-child process relationships
• Response: Process analysis + containment

Data Access Anomalies

• Pattern: Large file transfers outside business hours
• Pattern: Access to sensitive directories
• Response: Data loss prevention workflow

#### Windows特定检测

Event ID Monitoring

4624: Successful logon (track unusual locations/times) 4625: Failed logon (brute force detection) 4648: Explicit credential use (lateral movement) 4656: File/object access (data access monitoring) 4719: System audit policy changes (tampering detection) 4720: User account created (account management)

PowerShell Security

4103: Module logging (malicious command detection) 4104: Script block logging (obfuscated script analysis) 4105: Script start (execution monitoring) 4106: Script stop (completion tracking)

### 🔐 **身份验证和访问控制**

Create security roles

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_security/api/roles/security_analyst" \ -H 'Content-Type: application/json' \ -d '{ "cluster_permissions": ["cluster_composite_ops_ro"], "index_permissions": [{ "index_patterns": ["logstash-logs-*", "security-events-*"], "allowed_actions": ["read", "search"] }] }'

Create users with specific roles

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_security/api/internalusers/analyst1" \ -H 'Content-Type: application/json' \ -d '{ "password": "secure_password", "roles": ["security_analyst"], "attributes": { "department": "Security", "clearance_level": "L2" } }'

______________________________________________________________________

## 🔧 高级配置

### 高可用性设置

Add load balancer

nginx: image: nginx:alpine ports: - "80:80" - "443:443" volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf - ./nginx/ssl:/etc/nginx/ssl depends_on: - opensearch-node1 - opensearch-node2

Cross-cluster replication

opensearch-remote: image: opensearchproject/opensearch:latest environment: - cluster.name=opensearch-backup - replication.enabled=true

### 自定义日志处理管道

Enhanced logstash pipeline for Windows events

filter { if [agent][type] == "winlogbeat" { # Windows Event ID classification if [winlog][event_id] in [4624, 4634] { mutate { add_tag => ["authentication"] } } if [winlog][event_id] in [4625, 4648] { mutate { add_tag => ["failed_auth", "security_alert"] } } if [winlog][event_id] in [4656, 4658, 4660, 4663] { mutate { add_tag => ["file_access", "audit"] } }

# Extract user and computer information grok { match => { "[winlog][event_data][SubjectUserName]" => "(?[^$]+)" } tag_on_failure => ["_grok_user_failure"] }

# GeoIP enrichment for external IPs if [winlog][event_data][IpAddress] and [winlog][event_data][IpAddress] !~ /^(10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.)/ { geoip { source => "[winlog][event_data][IpAddress]" target => "geoip" database => "/usr/share/GeoIP/GeoLite2-City.mmdb" } } } }

Output to specialized indices

output { if "security_alert" in [tags] { opensearch { index => "security-events-%{+YYYY.MM.dd}" # ... connection details } } else if "authentication" in [tags] { opensearch { index => "auth-logs-%{+YYYY.MM.dd}" # ... connection details } } else { opensearch { index => "logstash-logs-%{+YYYY.MM.dd}" # ... connection details } } }

______________________________________________________________________

## 📊 监控和警报

### 健康监测

Service health checks

docker-compose exec opensearch-node1 curl -k \ "https://localhost:9200/_cluster/health?pretty"

Logstash pipeline monitoring

curl "http://localhost:9600/_node/stats/pipelines?pretty"

MCP server status

curl "http://localhost:8000/health"

### 性能指标

Index statistics

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=store.size:desc"

Search performance

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_nodes/stats/indices/search?pretty"

Memory usage

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu"

______________________________________________________________________

## 🔧 故障排除

### 常见问题

#### OpenSearch无法启动

Check logs

docker-compose logs opensearch-node1

Common fixes

sysctl -w vm.max_map_count=262144 echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

Verify disk space

df -h

#### Logstash处理问题

Check pipeline status

curl "http://localhost:9600/_node/stats/pipelines?pretty"

Monitor processing rates

docker-compose logs logstash | grep -E "(ERROR|WARN)"

Test configuration

docker-compose exec logstash logstash --config.test_and_exit

#### MCP服务器连接问题

Test MCP connectivity

curl http://localhost:8000/health

Check OpenSearch connectivity from MCP

docker-compose exec mcp-server python -c " from mcp_tools import OpenSearchClient client = OpenSearchClient('https://opensearch-node1:9200', 'admin', '${PASSWORD}') print(client.health_check()) "

### 日志分析命令

Find parsing failures

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/logstash-logs-*/_search?q=tags:_grokparsefailure"

Monitor ingestion rates

curl -k -u admin:${PASSWORD} \ "https://localhost:9200/_cat/indices/logstash-logs-*?v&s=docs.count:desc"

Check for duplicate events

curl -k -u admin:${PASSWORD} -X POST \ "https://localhost:9200/logstash-logs-*/_search" \ -d '{"aggs":{"duplicates":{"terms":{"field":"message.keyword","size":10}}}}'

______________________________________________________________________

## 🚀 生产部署

### 安全强化

SSL/TLS Configuration

opensearch-node1: environment: - plugins.security.ssl.http.enabled=true - plugins.security.ssl.http.pemcert_filepath=certs/node1.pem - plugins.security.ssl.http.pemkey_filepath=certs/node1-key.pem volumes: - ./certs:/usr/share/opensearch/config/certs:ro

Firewall rules (UFW example)

sudo ufw allow from TRUSTED_NETWORK to any port 9200 sudo ufw allow from TRUSTED_NETWORK to any port 5601 sudo ufw allow 514/tcp # Syslog sudo ufw allow 5044/tcp # Beats

### 备份和恢复

Snapshot configuration

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_snapshot/backup_repository" \ -H 'Content-Type: application/json' \ -d '{ "type": "fs", "settings": { "location": "/usr/share/opensearch/backups", "compress": true } }'

Automated daily snapshots

curl -k -u admin:${PASSWORD} -X PUT \ "https://localhost:9200/_plugins/_ism/policies/snapshot_policy" \ -H 'Content-Type: application/json' \ -d '{ "policy": { "description": "Daily snapshots", "default_state": "snapshot", "states": [{ "name": "snapshot", "actions": [{"snapshot": {"snapshot": "daily-{now/d}"}}], "transitions": [{"state_name": "snapshot", "conditions": {"cron": {"cron": {"expression": "0 2 * * *"}}}}] }] } }'

______________________________________________________________________

## 📚 文档和资源

### 📖 其他文件

- [🔧 高级Logstash配置](./docs/logstash-advanced.md)
- [🪟 Windows事件收集设置](./docs/windows-setup.md)
- [🐧 Linux日志源配置](./docs/linux-setup.md)
- [🛡️ 安全规则和警报](./docs/security-rules.md)
- [🚀 生产部署指南](./docs/production.md)
- [🎯 api参考](./docs/api-reference.md)

### 🔗 外部资源

- [OpenSearch文档](https://opensearch.org/docs/)
- [Logstash配置参考](https://www.elastic.co/guide/en/logstash/current/configuration.html)
- [FastMCP文档](https://github.com/jlowin/fastmcp)
- [MITRE ATT&CK框架](https://attack.mitre.org/)
- [SIGMA规则](https://github.com/SigmaHQ/sigma)

### 🎓 学习资源

- [OpenSearch查询DSL](https://opensearch.org/docs/latest/query-dsl/)
- [Grok模式测试](https://grokdebug.herokuapp.com/)
- [正则表达式测试](https://regex101.com/)

______________________________________________________________________

## 🤝 贡献

我们欢迎捐款！请阅读我们的 [贡献指南](CONTRIBUTING.md) 在提交PR之前。

### 开发设置

Clone repository

git clone https://github.com/your-username/opensearch-centralized-logs.git cd opensearch-centralized-logs

Setup development environment

cp .env_example .env docker-compose -f docker-compose.dev.yml up -d

Run tests

make test-all

Lint code

make lint

Generate documentation

make docs

### 🐛 问题模板

- **错误报告**：用于功能问题
- **功能请求**：建议新功能
- **安保问题**：私下报告安全漏洞
- **文档**：改进文档和示例

______________________________________________________________________

## 📄 许可证

此项目根据MIT许可证获得许可-请参阅 [许可证](LICENSE) 文件以获取详细信息。

______________________________________________________________________

## 🌟 支持与社区

- 📧 **问题**: 
- 💬 **讨论**: 
- 📖 **维基**: [维基工程](https://github.com/your-username/opensearch-centralized-logs/wiki)
- 🆘 **支持**: [社区论坛](https://github.com/your-username/opensearch-centralized-logs/discussions/categories/q-a)

### 🏆 贡献者

感谢我们出色的贡献者！看 [贡献者.md](CONTRIBUTORS.md) 查看完整列表。

______________________________________________________________________

## 🎯 路线图

### 📅 **2025年第一季度**

- \[\]增强的Windows事件解析
- \[\]基于机器学习的异常检测
- \[\]移动设备日志集成
- \[\]高级合规报告

### 📅 **2025年第二季度**

- \[\]Kubernetes日志收集
- \[\]云原生部署选项
- \[\]高级关联引擎
- \[\]自定义插件市场

### 📅 **2025年第3季度**

- \[\]多租户架构
- \[\]企业SSO集成
- \[\]高级威胁追踪工具
- \[\]自动事件响应

______________________________________________________________________

**⭐ 如果你觉得这个存储库有用，请为其加星！**

**🔗  | 📖 [文档](./docs/) | 🚀 [开始使用](#-quick-start)**

*内置于❤️ 面向网络安全和DevOps社区*