Features

Synthetic Heart is a highly modular AI system with extensible capabilities. The core system automatically discovers and integrates components at runtime, ensuring that functionality can be added or removed without code changes.

Modular Architecture

Component Auto-Discovery: synth automatically scans interface/, plugins/, and cortex/ directories to load compatible components. No manual registration or configuration files required.
Zero Hardcoding: Components are completely decoupled from the core. Adding new functionality requires only placing a compatible Python module in the appropriate directory.
Runtime Flexibility: Switch Cortex engines, enable/disable plugins, and manage interfaces dynamically without restarting the system.

Adaptive Intelligence

synth supports multiple language model backends with seamless switching. Note: Currently, only the Selenium ChatGPT (Legacy) engine is fully functional. Other engines are experimental and may not work reliably.

selenium_chatgpt_legacy – Legacy browser-controlled ChatGPT (fully functional).
selenium_chatgpt – Browser-controlled ChatGPT for advanced interaction (experimental).
selenium_gemini – Browser-controlled Google Gemini with multimodal support (experimental).
selenium_grok – Browser-controlled xAI Grok with reasoning capabilities (experimental).
manual – Human trainer input for debugging and development.

Runtime Engine Switching: Use /cortex <engine_name> to switch engines instantly during operation (/llm <engine_name> is a deprecated alias).

Multi-Platform Integration

Available Interfaces

Telegram: Bot support with media handling
Discord: Full bot integration with threading
Matrix: Chat bridge with real-time messaging
Ollama Compatible: REST API bridge for external clients

Cross-Platform Messaging

Send messages across different platforms using unified chat identifiers.

Progressive Web App (PWA) Mode (Experimental)

Status: In development — may not be fully functional. Use with caution in production environments.

The Web UI can be installed as a Progressive Web App (PWA) on supported browsers to provide an app-like experience (fullscreen, home-screen install, and basic offline caching). A service worker is included (see res/synth_webui/static/service-worker.js) and the Web UI templates register the service worker and handle the beforeinstallprompt event to show an install button.

Known limitations include partial offline behavior (real-time features are limited while offline), varying support for push notifications and background sync across browsers, and occasional install prompt availability depending on the client environment.

Desktop Notifications (Web UI)

The Web UI can display in-browser desktop notifications when new messages arrive while the tab is in the background.

Notes:

Requires HTTPS (or localhost) due to browser security restrictions.
The browser permission prompt is requested on the first user action that sends a message.
Notifications can be toggled from the Web UI settings.

See docs/chat_history.rst for how session persistence and archives interact with the Web UI when using PWA mode.

Web UI Chat Archiving

The Web UI supports archiving and restoring entire conversations for the single persistent session. Archives are filesystem-backed JSON snapshots stored under backups/chat_archives/ and exposed via the Web UI API. Basic operations (archive, list, restore, delete) are available and documented in Chat History System.

This archiving mechanism is intended for the Web UI MVP; production deployments may prefer database-backed archives or additional metadata storage.

Avatar System

VRM Avatar Support: Synthetic Heart includes a default SyntH avatar for 3D visualization, but users can provide their own VRM avatar files for customization. The system supports animated states including idle, talking, and thinking, reflecting the persona’s current activity.

Web UI Animation Sync

The Web UI receives real-time animation commands from the backend. When the global action phase changes (e.g. THINKING, WRITING, TALKING, IDLE), the backend triggers the AnimationHandler to broadcast the corresponding animation state to Web UI clients. This keeps the backend authoritative for animation timing, while still allowing plugins/skins to override which files are used for each logical state.

Plugin Ecosystem

Action Plugins

ai_diary: Personal memory and interaction tracking
bio_manager: Persistent user profile management
blocklist: User access control
chat_link: Conversation linking and management
message_map: Message threading and tracking
message_plugin: Cross-platform message routing
recent_chats: Access to conversation history
time_plugin: Time and location awareness
weather_plugin: Real-time weather information

Contextual Memory

Message Context: Toggle contextual memory with /context on/off to include recent conversation history in prompts.
Personal Memory: The AI Diary plugin provides persistent memory of interactions, emotions, and relationships.
User Profiles: Bio management system maintains detailed user profiles for personalized interactions.

Security & Access Control

Trainer ID Validation: Sensitive operations require authorization from configured trainer IDs.
Rate Limiting: Built-in rate limiting prevents abuse across all components.
Input Validation: All actions are validated against component schemas before execution.
Error Handling: Comprehensive error handling with user-friendly notifications and automatic recovery.

Extensibility

Creating New Components

Add functionality by implementing AIPluginBase and placing modules in appropriate directories:

Plugins: Extend capabilities with new actions
Cortex engines: Add support for new runtime engines (kinds: llm, live, agent).
Interfaces: Integrate new communication platforms

No Core Modifications

Components are self-contained and register their capabilities automatically. The core system remains unchanged when adding features.

Development Friendly

Clear interfaces, comprehensive documentation, and example implementations make extension straightforward.

AI Diary

The AI Diary is a modular plugin that provides synth with persistent memory of interactions and activities. This plugin is completely self-contained and can be removed without affecting the core system.

Key Features:

Modular Design: The diary plugin is fully self-contained with internal configuration and dedicated database storage.
Automatic Entry Creation: After each action execution, the system creates diary entries summarizing activities, involved parties, tags, and emotions.
Static Injection: Recent diary entries are injected into prompts when space allows, providing context from previous interactions.
User Access: Authorized users (trainers) can view diary entries using the /diary command.
Fail-Safe Operation: The plugin automatically disables itself in case of errors, ensuring the core system continues functioning.

Database Schema:

CREATE TABLE ai_diary (
    id INT AUTO_INCREMENT PRIMARY KEY,
    content TEXT NOT NULL,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    tags JSON,
    involved JSON,
    emotions JSON,
    interface_path VARCHAR(512),
    INDEX idx_timestamp (timestamp),
    INDEX idx_interface_path (interface_path)
);

Usage Example:

After helping Takeshi with a coding task, synth automatically creates a diary entry:

=== synth's Recent Diary ===

📅 2024-01-15 14:30:22
Helped Takeshi with bio update and security improvements
#tags: bio, security, helpful
#involved: Takeshi
#emotions: helpful(8), focused(7)
#context: telegram_bot/123456/2

=== End Diary ===

Plugin Management:

The diary plugin can be enabled/disabled dynamically:

from plugins.ai_diary import is_plugin_enabled, enable_plugin, disable_plugin

# Check status
if is_plugin_enabled():
    print("Plugin active")

# Disable manually
disable_plugin()

# Re-enable (tests database connection)
success = enable_plugin()

Configuration:

Each Cortex engine has its own configuration for diary integration:

OpenAI: Up to 2000 characters for diary content
Selenium ChatGPT: Up to 1500 characters
Google CLI: Up to 1200 characters
Manual: Up to 800 characters

This ensures optimal performance across different interfaces while maintaining contextual awareness.