# Surface Painter: Multi-Modal Painting for Interactive Spaces ## Shape Up Pitch for AnySurface Platform --- ## Problem **Creative expression in physical spaces is limited by traditional tools and surfaces.** People want to paint, annotate, and create directly on walls, tables, floors, and any surface in their environment, but current solutions are either: - **Too restrictive**: Limited to specific hardware (smart boards, tablets) - **Too complex**: Require extensive setup or technical knowledge - **Not collaborative**: Can't easily share creative control between local and remote users - **Single-modal**: Force users into one interaction method instead of natural, contextual choices **Why now?** The AnySurface platform provides the foundational infrastructure (calibrated projectors, depth mapping, surface detection) that makes true spatial painting possible. We have the technical capability to turn any room into a creative canvas - we need the application layer to make it meaningful. **Current alternatives:** - Digital whiteboards: Expensive, fixed-position, single-surface - Tablet/stylus painting: Limited to small screens, not collaborative - AR drawing apps: Require headsets, limited persistence, single-user - Projection mapping: Complex setup, not interactive, requires specialized knowledge --- ## Appetite **6-week cycle** - This is a foundational creative tool that will demonstrate the full potential of the AnySurface platform. It's worth significant investment to get right. **Trade-offs we're willing to make:** - Start with basic brush tools before advanced artistic features - Focus on core interaction modes before exotic input methods - Prioritize reliability over feature richness in v1 - Accept some calibration complexity for multi-projector setups --- ## Solution ### Core Concept **Surface Painter transforms any room into a collaborative digital canvas** using multiple interaction modalities that adapt to user context and capability. A person can paint with a stylus while someone else finger-paints on the wall while a third person paints remotely via video feed. ### Key Components #### 1. **Universal Canvas Engine** - HTML5 canvas that scales across multiple projected surfaces - Coordinate system that maps between physical space, projector space, and canvas space - Real-time synchronization across all interaction modalities - Persistent drawings that survive app restarts and room reconfigurations #### 2. **Multi-Modal Input System** Four distinct interaction modes that can operate simultaneously: **Stylus/Hover Mode (Current)** - Android device casting to projector - Stylus hover detection for precise cursor positioning - Touch-to-paint with pressure sensitivity - Ideal for detailed work and technical drawing **Remote Painting Mode** - WebRTC camera feed from room to remote user - Point-and-click painting on video stream - Projector correspondence mapping to translate video coordinates to physical space - Enables remote collaboration and accessibility **Laser Pointer Mode** - Computer vision tracking of laser dot position - Click detection via laser pattern recognition or separate input - Natural pointing gesture familiar to presenters - Works across large surfaces and distances **Finger Painting Mode** - Hand/finger tracking via depth cameras - Direct touch detection on projected surfaces - Multi-user simultaneous painting - Intuitive for casual interaction and children #### 3. **Adaptive Interface System** - Tool palette that follows the active user - Context-sensitive controls based on interaction mode - Minimizable interface that stays out of the way - Fullscreen mode for maximum canvas space #### 4. **Cross-Modal Features** - Dwell-time activation for hands-free painting - Preview modes for precise placement - Pressure/velocity sensitivity across input types - Shared color palette and brush settings ### Connecting the Dots The AnySurface platform provides: - **Surface detection** → Canvas boundaries and projection mapping - **Calibrated projectors** → Coordinate transformation and multi-projector alignment - **Depth mapping** → Touch detection and occlusion handling - **Camera feeds** → Remote collaboration and finger tracking Surface Painter adds: - **Unified coordinate system** across all input modalities - **Real-time rendering** that works across projected surfaces - **Collaboration layer** for simultaneous multi-user interaction - **Persistence layer** for saving and restoring artwork --- ## Rabbit Holes ### Known Technical Risks **Coordinate transformation complexity**: Converting between camera space, projector space, and canvas space across multiple projectors could create accuracy issues - *Mitigation*: Start with single-projector setups, build robust calibration tools **Real-time performance**: Multiple input streams plus rendering could cause latency - *Mitigation*: Profile early, optimize coordinate calculations, use WebGL where beneficial **WebRTC streaming quality**: Remote painting depends on low-latency, high-quality video - *Mitigation*: Implement adaptive quality, test with various network conditions **Multi-user conflict resolution**: Simultaneous painting could create inconsistent state - *Mitigation*: Use operational transforms or last-writer-wins with clear visual feedback ### Cut-Outs (Explicitly NOT doing) - **Advanced artistic tools**: No vectors, layers, or complex brushes in v1 - **3D painting**: Stick to 2D surfaces initially - **Voice control**: Not including audio input modalities - **Gesture recognition**: Beyond basic finger pointing, no complex gestures - **File import/export**: No external image integration in v1 ### No-Gos - **No single-point-of-failure**: Each input mode must work independently - **No expensive hardware requirements**: Should work with standard cameras and projectors - **No platform lock-in**: Must work across different devices and browsers - **No data loss**: Paintings must survive technical failures --- ## No-Brainers **Background color selection**: Every painting tool needs this basic feature **Clear canvas function**: Essential for starting over **Basic brush size control**: Core functionality for any painting app **Fullscreen toggle**: Maximize canvas space for projection scenarios **Undo functionality**: Standard expectation for creative tools **Save/load capability**: Persistence is table stakes for creative work --- ## Success Metrics **Technical Success:** - Sub-100ms latency between input and visual feedback - Accurate coordinate mapping across all input modes - Stable multi-user collaboration without conflicts **User Success:** - Natural adoption of different input modes based on context - Successful remote collaboration sessions - Creative output that leverages unique spatial capabilities **Platform Success:** - Demonstrates AnySurface's creative potential to new audiences - Provides template for other spatial HTML applications - Showcases seamless hardware/software integration --- ## Implementation Phases **Week 1-2**: Core canvas engine and coordinate system foundation **Week 3-4**: Stylus mode refinement and remote painting integration **Week 5-6**: Laser pointer and finger tracking integration, multi-modal coordination This positions Surface Painter as the flagship creative application for AnySurface, demonstrating how multiple interaction modalities can seamlessly blend in a spatial computing environment.