shellx-ai
Version:
shellx is a powerful WebSocket-based client for controlling shell commands and UI automation on remote devices.
432 lines (327 loc) • 13.8 kB
Markdown
# ShellX
[](https://badge.fury.io/js/shellx)
[](https://opensource.org/licenses/MIT)
**ShellX** is a powerful automation framework for Android device control and UI automation. It provides seamless shell command execution, element finding, screen capture, and automated task execution through an intelligent ShellX.ai service architecture.
## ✨ Features
- 🔐 **Zero-Configuration Setup** - Automatic ShellX.ai authentication and service connection
- 📱 **Android Device Control** - Execute shell commands with real-time output monitoring
- 🎯 **Smart UI Automation** - Advanced element finding with retry logic and multiple strategies
- 📸 **Screen Operations** - Screenshots, screen info, and visual change detection
- 🔄 **Intelligent Fallback** - Automatic switching between ShellX.ai cloud and local services
- 🛡️ **Robust Error Handling** - Comprehensive error recovery and diagnostics
- 🧠 **Shell Output Monitoring** - Real-time command output processing and analysis
- 📊 **Rich Diagnostics** - Built-in element analysis and debugging tools
- 🔗 **Smart Connection Management** - ShellX.ai service connection status with message-based verification
## 🚀 Quick Start
### Installation
```bash
npm install shellx-ai
```
### Node.js Version Compatibility
ShellX supports Node.js 14+ with automatic polyfills:
**HTTP Requests:**
- **Fallback**: Node.js 18+ built-in `fetch` or `node-fetch` for older versions
- **Browser**: Compatible with all fetch implementations
**ShellX.ai Service Connection:**
- **Browser**: Uses native `WebSocket` API for service communication
- **Node.js**: Automatically uses `ws` module for service communication
No additional configuration needed - ShellX handles environment detection automatically!
### Basic Setup
```typescript
import { createShellXWithShellMonitoring } from 'shellx';
// Set your authentication key
process.env.SHELLX_AUTH_KEY = 'your-auth-key';
// Initialize ShellX with automatic ShellX.ai authentication
const { client, shellx } = await createShellXWithShellMonitoring();
// Start automating!
await shellx.executeShellCommand('getprop ro.build.version.release');
await shellx.clickByText('Settings');
```
### Environment Setup
Set your authentication key as an environment variable:
```bash
# Linux/macOS
export SHELLX_AUTH_KEY="your-auth-key"
# Windows
set SHELLX_AUTH_KEY=your-auth-key
```
## 📱 Common Use Cases
### 1. Shell Command Execution
```typescript
// Execute shell commands with real-time output
const result = await shellx.executeShellCommand('pm list packages', {
title: 'List installed packages',
timeout: 10000,
allowPartialResult: true, // Return partial results on timeout instead of throwing error
onOutput: (output) => {
console.log('Real-time output:', output);
}
});
console.log('Command result:', result.output);
console.log('Success:', result.success);
console.log('Duration:', result.duration, 'ms');
```
### 2. UI Element Operations
```typescript
// Smart element finding with retry logic
const elements = await shellx.findElementsWithRetry({
className: 'android.widget.TextView',
textContains: 'Settings'
}, 3, 1000, {
maxResults: 10,
visibleOnly: true
});
// Print detailed element information
shellx.printElementsInfo(elements, 'Found Settings Elements');
// Click by text content
await shellx.clickByText('Wi-Fi', false);
// Input text with options
await shellx.inputText({
resourceId: 'search_field'
}, 'Hello ShellX', {
clear: true,
hideKeyboard: true
});
```
### 3. Screen Operations
```typescript
// Capture screenshot with details
const screenshot = await shellx.captureScreen({
format: 'png',
quality: 90,
saveInfo: true
});
// Swipe gestures
await shellx.swipe('up', 400, 800); // direction, distance, duration
await shellx.swipe('left', 300, 600);
```
### 4. App Automation Workflow
```typescript
// Login automation example
await shellx.performLogin(
{ resourceId: 'username_field' }, // username selector
{ resourceId: 'password_field' }, // password selector
{ resourceId: 'login_button' }, // login button selector
'user@example.com', // username
'password123' // password
);
// Navigation workflow
await shellx.navigateByPath([
'Settings',
'Network & internet',
'Wi-Fi'
]);
```
### 5. Advanced Element Finding
```typescript
// Wait for any of multiple elements to appear
const result = await shellx.waitForAnyElement([
{ textContains: 'OK' },
{ textContains: 'Cancel' },
{ textContains: 'Skip' }
], 10000);
if (result) {
console.log(`Found element: ${result.element.text} at index ${result.selectorIndex}`);
}
// Scroll to find element
const element = await shellx.scrollToFindElement({
textContains: 'Privacy Settings'
}, 5, 'down');
```
### 6. Batch Shell Commands
```typescript
// Execute multiple commands in sequence
const commands = [
{ command: 'am force-stop com.example.app', title: 'Stop app' },
{ command: 'am start -n com.example.app/.MainActivity', title: 'Start app' },
{ command: 'input keyevent KEYCODE_HOME', title: 'Go home', waitAfterMs: 1000 }
];
const results = await shellx.executeShellCommands(commands, {
continueOnError: true,
timeout: 5000
});
```
### 7. Device Information
```typescript
// Get comprehensive device information
const deviceInfo = await shellx.getDeviceInfo();
deviceInfo.forEach((info, index) => {
console.log(`${index + 1}. ${info.output}`);
});
// ADB commands with enhanced output
const result = await shellx.adbCommand('shell dumpsys battery', {
title: 'Get battery status',
successPattern: /level: \d+/,
errorPattern: /error|failed/i
});
```
## 🎯 ShellX Class API
### Core Methods
| Method | Description | Example |
|--------|-------------|---------|
| `executeShellCommand(cmd, opts?)` | Execute shell command with monitoring | `shellx.executeShellCommand('ls -la', {allowPartialResult: true})` |
| `completeTask(taskId, result?, success?)` | Manually complete a specific task | `client.completeTask('task-123', {output: 'result'})` |
| `getPendingTaskIds()` | Get all pending task IDs | `client.getPendingTaskIds()` |
| `getTaskInfo(taskId)` | Get task information | `client.getTaskInfo('task-123')` |
| `completeTasksByType(type, result?, success?)` | Complete all tasks of a specific type | `client.completeTasksByType('action')` |
| `clickByText(text, exact?)` | Click element by text content | `shellx.clickByText('OK', true)` |
| `inputText(selector, text, opts?)` | Input text into element | `shellx.inputText({resourceId: 'field'}, 'text')` |
| `findElementWithRetry(selector, retries?, delay?)` | Find single element with retry | `shellx.findElementWithRetry({text: 'Button'})` |
| `findElementsWithRetry(selector, retries?, delay?, opts?)` | Find multiple elements with retry | `shellx.findElementsWithRetry({className: 'TextView'})` |
| `swipe(direction, distance?, duration?)` | Perform swipe gesture | `shellx.swipe('up', 400, 800)` |
| `captureScreen(opts?)` | Take screenshot | `shellx.captureScreen({saveInfo: true})` |
### Utility Methods
| Method | Description |
|--------|-------------|
| `printElementInfo(element, index?)` | Print detailed element information |
| `printElementsInfo(elements, title?)` | Print information for multiple elements |
| `waitForAnyElement(selectors, timeout?)` | Wait for any of multiple elements |
| `scrollToFindElement(selector, maxScrolls?, direction?)` | Scroll to find element |
| `performLogin(userSel, passSel, btnSel, user, pass)` | Automated login flow |
| `navigateByPath(textPath)` | Navigate through UI by text path |
| `getDeviceInfo()` | Get comprehensive device information |
## 🛠️ Configuration
### Environment Variables
```bash
# Required: Your ShellX authentication key
SHELLX_AUTH_KEY=your-auth-key
# Optional: Timeout settings
SHELLX_TIMEOUT=20000
SHELLX_RETRY_ATTEMPTS=3
```
### Connection Options
```typescript
const { client, shellx } = await createShellXWithShellMonitoring({
timeout: 15000,
reconnectMaxAttempts: 3,
onOpen: () => console.log('Connected'),
onClose: (event) => console.log('Disconnected'),
onError: (error) => console.error('Error:', error),
onMessage: (data) => {
// Handle custom messages
}
});
```
## 🔧 Advanced Features
### Shell Command Options
ShellX provides flexible options for shell command execution:
```typescript
const result = await shellx.executeShellCommand('long-running-command', {
title: 'Custom command title',
timeout: 30000, // Command timeout in milliseconds
allowPartialResult: true, // Return partial results on timeout instead of throwing error
waitAfterMs: 1000, // Wait time after command completion
onOutput: (output) => { // Real-time output callback
console.log('Live output:', output);
},
onError: (error) => { // Error callback
console.error('Command error:', error);
},
successPattern: /success|completed/, // Pattern to detect success
errorPattern: /error|failed/ // Pattern to detect failure
});
```
#### allowPartialResult Parameter
The `allowPartialResult` parameter controls timeout behavior:
- **`true`**: When command times out but has partial output, returns the partial result instead of throwing an error
- **`false`**: When command times out, always throws an error (default behavior)
- **`undefined`**: Uses default behavior (returns partial results if available)
```typescript
// Example: Allow partial results for long-running commands
const result = await shellx.executeShellCommand('find / -name "*.log"', {
timeout: 5000,
allowPartialResult: true
});
if (!result.success && result.error?.includes('timeout')) {
console.log('Command timed out, but got partial results:', result.output);
} else {
console.log('Command completed successfully:', result.output);
}
```
### Task Management
ShellX provides advanced task management capabilities for manual control:
```typescript
// Send message and get taskId for manual control
const { taskId, promise } = await client.sendMessageWithTaskId(
{ actions: { type: 'command', command: 'long-running-command' } },
'action',
'command'
);
console.log('Task ID:', taskId);
// Manually complete the task when ready
client.completeTask(taskId, { output: 'Command completed' }, true);
// Get all pending task IDs
const pendingTasks = client.getPendingTaskIds();
console.log('Pending tasks:', pendingTasks);
// Get task information
const taskInfo = client.getTaskInfo(taskId);
console.log('Task info:', taskInfo);
// Complete all tasks of a specific type
const completedCount = client.completeTasksByType('action', { success: true });
console.log(`Completed ${completedCount} action tasks`);
```
### Shell Output Monitoring
ShellX automatically monitors shell command output in real-time:
```typescript
const result = await shellx.executeShellCommand('top -n 1', {
onOutput: (output) => {
console.log('Live output:', output);
},
successPattern: /Tasks:/,
errorPattern: /error|failed/i,
timeout: 10000,
allowPartialResult: true // Return partial results on timeout
});
```
### Element Diagnostics
Built-in diagnostic tools for troubleshooting:
```typescript
// Automatic element diagnosis when finding fails
const elements = await shellx.findElementsWithRetry({
textContains: 'NotFound'
});
if (elements.length === 0) {
// ShellX automatically runs diagnostics
// Shows available elements, device info, etc.
}
```
### Intelligent Fallback
ShellX automatically handles service availability:
```typescript
// 1. Tries online authentication: https://shellx.ai/api/device/{authKey}
// 2. On success: Uses returned machine URL
// 3. On failure: Falls back to local: ws://127.0.0.1:9091/api/s/{authKey}
// 4. Seamless operation regardless of network conditions
```
## 📚 Complete Examples
Check out the [examples directory](./examples/) for comprehensive use cases:
- **[basic.ts](./examples/basic.ts)** - Basic automation operations
- **[shell-commands-demo.ts](./examples/shell-commands-demo.ts)** - Shell command execution
- **[find-elements-demo.ts](./examples/find-elements-demo.ts)** - Element finding strategies
## 🚨 Error Handling
ShellX provides comprehensive error handling with automatic recovery:
```typescript
try {
await shellx.executeShellCommand('invalid-command');
} catch (error) {
// ShellX handles timeouts, connection issues, and command failures
console.error('Command failed:', error.message);
// Automatic diagnostics on failure
// Device info, available elements, etc.
}
```
## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## 📄 License
MIT License - see the [LICENSE](LICENSE) file for details.
## 🆘 Support
- 📚 [Documentation](https://shellx.ai/docs)
- 🐛 [GitHub Issues](https://github.com/your-org/shellx/issues)
- 💬 [Discord Community](https://discord.gg/shellx)
## 🎉 Getting Started
1. **Install ShellX**: `npm install shellx-ai`
2. **Set your auth key**: `export SHELLX_AUTH_KEY="your-key"`
3. **Run the example**: `ts-node examples/basic.ts`
4. **Start automating!** 🚀
---
*Made with ❤️ for the automation community*