UNPKG

shellx-ai

Version:

shellx is a powerful WebSocket-based client for controlling shell commands and UI automation on remote devices.

432 lines (327 loc) 13.8 kB
# ShellX [![npm version](https://badge.fury.io/js/shellx.svg)](https://badge.fury.io/js/shellx) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) **ShellX** is a powerful automation framework for Android device control and UI automation. It provides seamless shell command execution, element finding, screen capture, and automated task execution through an intelligent ShellX.ai service architecture. ## ✨ Features - 🔐 **Zero-Configuration Setup** - Automatic ShellX.ai authentication and service connection - 📱 **Android Device Control** - Execute shell commands with real-time output monitoring - 🎯 **Smart UI Automation** - Advanced element finding with retry logic and multiple strategies - 📸 **Screen Operations** - Screenshots, screen info, and visual change detection - 🔄 **Intelligent Fallback** - Automatic switching between ShellX.ai cloud and local services - 🛡️ **Robust Error Handling** - Comprehensive error recovery and diagnostics - 🧠 **Shell Output Monitoring** - Real-time command output processing and analysis - 📊 **Rich Diagnostics** - Built-in element analysis and debugging tools - 🔗 **Smart Connection Management** - ShellX.ai service connection status with message-based verification ## 🚀 Quick Start ### Installation ```bash npm install shellx-ai ``` ### Node.js Version Compatibility ShellX supports Node.js 14+ with automatic polyfills: **HTTP Requests:** - **Fallback**: Node.js 18+ built-in `fetch` or `node-fetch` for older versions - **Browser**: Compatible with all fetch implementations **ShellX.ai Service Connection:** - **Browser**: Uses native `WebSocket` API for service communication - **Node.js**: Automatically uses `ws` module for service communication No additional configuration needed - ShellX handles environment detection automatically! ### Basic Setup ```typescript import { createShellXWithShellMonitoring } from 'shellx'; // Set your authentication key process.env.SHELLX_AUTH_KEY = 'your-auth-key'; // Initialize ShellX with automatic ShellX.ai authentication const { client, shellx } = await createShellXWithShellMonitoring(); // Start automating! await shellx.executeShellCommand('getprop ro.build.version.release'); await shellx.clickByText('Settings'); ``` ### Environment Setup Set your authentication key as an environment variable: ```bash # Linux/macOS export SHELLX_AUTH_KEY="your-auth-key" # Windows set SHELLX_AUTH_KEY=your-auth-key ``` ## 📱 Common Use Cases ### 1. Shell Command Execution ```typescript // Execute shell commands with real-time output const result = await shellx.executeShellCommand('pm list packages', { title: 'List installed packages', timeout: 10000, allowPartialResult: true, // Return partial results on timeout instead of throwing error onOutput: (output) => { console.log('Real-time output:', output); } }); console.log('Command result:', result.output); console.log('Success:', result.success); console.log('Duration:', result.duration, 'ms'); ``` ### 2. UI Element Operations ```typescript // Smart element finding with retry logic const elements = await shellx.findElementsWithRetry({ className: 'android.widget.TextView', textContains: 'Settings' }, 3, 1000, { maxResults: 10, visibleOnly: true }); // Print detailed element information shellx.printElementsInfo(elements, 'Found Settings Elements'); // Click by text content await shellx.clickByText('Wi-Fi', false); // Input text with options await shellx.inputText({ resourceId: 'search_field' }, 'Hello ShellX', { clear: true, hideKeyboard: true }); ``` ### 3. Screen Operations ```typescript // Capture screenshot with details const screenshot = await shellx.captureScreen({ format: 'png', quality: 90, saveInfo: true }); // Swipe gestures await shellx.swipe('up', 400, 800); // direction, distance, duration await shellx.swipe('left', 300, 600); ``` ### 4. App Automation Workflow ```typescript // Login automation example await shellx.performLogin( { resourceId: 'username_field' }, // username selector { resourceId: 'password_field' }, // password selector { resourceId: 'login_button' }, // login button selector 'user@example.com', // username 'password123' // password ); // Navigation workflow await shellx.navigateByPath([ 'Settings', 'Network & internet', 'Wi-Fi' ]); ``` ### 5. Advanced Element Finding ```typescript // Wait for any of multiple elements to appear const result = await shellx.waitForAnyElement([ { textContains: 'OK' }, { textContains: 'Cancel' }, { textContains: 'Skip' } ], 10000); if (result) { console.log(`Found element: ${result.element.text} at index ${result.selectorIndex}`); } // Scroll to find element const element = await shellx.scrollToFindElement({ textContains: 'Privacy Settings' }, 5, 'down'); ``` ### 6. Batch Shell Commands ```typescript // Execute multiple commands in sequence const commands = [ { command: 'am force-stop com.example.app', title: 'Stop app' }, { command: 'am start -n com.example.app/.MainActivity', title: 'Start app' }, { command: 'input keyevent KEYCODE_HOME', title: 'Go home', waitAfterMs: 1000 } ]; const results = await shellx.executeShellCommands(commands, { continueOnError: true, timeout: 5000 }); ``` ### 7. Device Information ```typescript // Get comprehensive device information const deviceInfo = await shellx.getDeviceInfo(); deviceInfo.forEach((info, index) => { console.log(`${index + 1}. ${info.output}`); }); // ADB commands with enhanced output const result = await shellx.adbCommand('shell dumpsys battery', { title: 'Get battery status', successPattern: /level: \d+/, errorPattern: /error|failed/i }); ``` ## 🎯 ShellX Class API ### Core Methods | Method | Description | Example | |--------|-------------|---------| | `executeShellCommand(cmd, opts?)` | Execute shell command with monitoring | `shellx.executeShellCommand('ls -la', {allowPartialResult: true})` | | `completeTask(taskId, result?, success?)` | Manually complete a specific task | `client.completeTask('task-123', {output: 'result'})` | | `getPendingTaskIds()` | Get all pending task IDs | `client.getPendingTaskIds()` | | `getTaskInfo(taskId)` | Get task information | `client.getTaskInfo('task-123')` | | `completeTasksByType(type, result?, success?)` | Complete all tasks of a specific type | `client.completeTasksByType('action')` | | `clickByText(text, exact?)` | Click element by text content | `shellx.clickByText('OK', true)` | | `inputText(selector, text, opts?)` | Input text into element | `shellx.inputText({resourceId: 'field'}, 'text')` | | `findElementWithRetry(selector, retries?, delay?)` | Find single element with retry | `shellx.findElementWithRetry({text: 'Button'})` | | `findElementsWithRetry(selector, retries?, delay?, opts?)` | Find multiple elements with retry | `shellx.findElementsWithRetry({className: 'TextView'})` | | `swipe(direction, distance?, duration?)` | Perform swipe gesture | `shellx.swipe('up', 400, 800)` | | `captureScreen(opts?)` | Take screenshot | `shellx.captureScreen({saveInfo: true})` | ### Utility Methods | Method | Description | |--------|-------------| | `printElementInfo(element, index?)` | Print detailed element information | | `printElementsInfo(elements, title?)` | Print information for multiple elements | | `waitForAnyElement(selectors, timeout?)` | Wait for any of multiple elements | | `scrollToFindElement(selector, maxScrolls?, direction?)` | Scroll to find element | | `performLogin(userSel, passSel, btnSel, user, pass)` | Automated login flow | | `navigateByPath(textPath)` | Navigate through UI by text path | | `getDeviceInfo()` | Get comprehensive device information | ## 🛠️ Configuration ### Environment Variables ```bash # Required: Your ShellX authentication key SHELLX_AUTH_KEY=your-auth-key # Optional: Timeout settings SHELLX_TIMEOUT=20000 SHELLX_RETRY_ATTEMPTS=3 ``` ### Connection Options ```typescript const { client, shellx } = await createShellXWithShellMonitoring({ timeout: 15000, reconnectMaxAttempts: 3, onOpen: () => console.log('Connected'), onClose: (event) => console.log('Disconnected'), onError: (error) => console.error('Error:', error), onMessage: (data) => { // Handle custom messages } }); ``` ## 🔧 Advanced Features ### Shell Command Options ShellX provides flexible options for shell command execution: ```typescript const result = await shellx.executeShellCommand('long-running-command', { title: 'Custom command title', timeout: 30000, // Command timeout in milliseconds allowPartialResult: true, // Return partial results on timeout instead of throwing error waitAfterMs: 1000, // Wait time after command completion onOutput: (output) => { // Real-time output callback console.log('Live output:', output); }, onError: (error) => { // Error callback console.error('Command error:', error); }, successPattern: /success|completed/, // Pattern to detect success errorPattern: /error|failed/ // Pattern to detect failure }); ``` #### allowPartialResult Parameter The `allowPartialResult` parameter controls timeout behavior: - **`true`**: When command times out but has partial output, returns the partial result instead of throwing an error - **`false`**: When command times out, always throws an error (default behavior) - **`undefined`**: Uses default behavior (returns partial results if available) ```typescript // Example: Allow partial results for long-running commands const result = await shellx.executeShellCommand('find / -name "*.log"', { timeout: 5000, allowPartialResult: true }); if (!result.success && result.error?.includes('timeout')) { console.log('Command timed out, but got partial results:', result.output); } else { console.log('Command completed successfully:', result.output); } ``` ### Task Management ShellX provides advanced task management capabilities for manual control: ```typescript // Send message and get taskId for manual control const { taskId, promise } = await client.sendMessageWithTaskId( { actions: { type: 'command', command: 'long-running-command' } }, 'action', 'command' ); console.log('Task ID:', taskId); // Manually complete the task when ready client.completeTask(taskId, { output: 'Command completed' }, true); // Get all pending task IDs const pendingTasks = client.getPendingTaskIds(); console.log('Pending tasks:', pendingTasks); // Get task information const taskInfo = client.getTaskInfo(taskId); console.log('Task info:', taskInfo); // Complete all tasks of a specific type const completedCount = client.completeTasksByType('action', { success: true }); console.log(`Completed ${completedCount} action tasks`); ``` ### Shell Output Monitoring ShellX automatically monitors shell command output in real-time: ```typescript const result = await shellx.executeShellCommand('top -n 1', { onOutput: (output) => { console.log('Live output:', output); }, successPattern: /Tasks:/, errorPattern: /error|failed/i, timeout: 10000, allowPartialResult: true // Return partial results on timeout }); ``` ### Element Diagnostics Built-in diagnostic tools for troubleshooting: ```typescript // Automatic element diagnosis when finding fails const elements = await shellx.findElementsWithRetry({ textContains: 'NotFound' }); if (elements.length === 0) { // ShellX automatically runs diagnostics // Shows available elements, device info, etc. } ``` ### Intelligent Fallback ShellX automatically handles service availability: ```typescript // 1. Tries online authentication: https://shellx.ai/api/device/{authKey} // 2. On success: Uses returned machine URL // 3. On failure: Falls back to local: ws://127.0.0.1:9091/api/s/{authKey} // 4. Seamless operation regardless of network conditions ``` ## 📚 Complete Examples Check out the [examples directory](./examples/) for comprehensive use cases: - **[basic.ts](./examples/basic.ts)** - Basic automation operations - **[shell-commands-demo.ts](./examples/shell-commands-demo.ts)** - Shell command execution - **[find-elements-demo.ts](./examples/find-elements-demo.ts)** - Element finding strategies ## 🚨 Error Handling ShellX provides comprehensive error handling with automatic recovery: ```typescript try { await shellx.executeShellCommand('invalid-command'); } catch (error) { // ShellX handles timeouts, connection issues, and command failures console.error('Command failed:', error.message); // Automatic diagnostics on failure // Device info, available elements, etc. } ``` ## 🤝 Contributing Contributions are welcome! Please feel free to submit a Pull Request. ## 📄 License MIT License - see the [LICENSE](LICENSE) file for details. ## 🆘 Support - 📚 [Documentation](https://shellx.ai/docs) - 🐛 [GitHub Issues](https://github.com/your-org/shellx/issues) - 💬 [Discord Community](https://discord.gg/shellx) ## 🎉 Getting Started 1. **Install ShellX**: `npm install shellx-ai` 2. **Set your auth key**: `export SHELLX_AUTH_KEY="your-key"` 3. **Run the example**: `ts-node examples/basic.ts` 4. **Start automating!** 🚀 --- *Made with ❤️ for the automation community*