# Redfish Creed Our basic tenets for an Architecture for Complexity ## 1. Data is the Protocol: datasets Models goes to die ### Dual Field Theory - Least Action Path - Micro Asymmetry of interaction rules as universal control parameter for macroscopic phase transitions in symmetry breaking ## 2. The Browser is the OS The browser manages data storage, networking, executing applications, controlling peripherals and window layering. In many ways this is the definition of an operating system. **Browser-First strategy** is keeping code, simulations and data management in the local browser. There will be cases where we host data, processes or data translation in servers to manage functions but we will try to replace these "leakages" as we move forward. Decentralized modules will publish and subscribe datasets. Avoid relying on remote procedure calls. In the case of mobile agents, treat their script as data as well. ## 3. URLs as API channels (Acequia) A Firebase Reference is a URL is our internal protocol (network and data format) that allows our models/apps to bind via publish/subscribe with permissions. The Reference can be a single field or a full hierarchy in a JSON tree. Note that these channels are language, OS and Client/Server agnostic. Our code "modules" can run in any location or language and participate in the Acequia. ## 4. Architect for Synchronized State not Message Passing Our first instinct tends to send messages over a channel to coordinate distributed programs. We should abstract away those messages and architect for a distributed state and listen for their changes. Anant Narayanan from Firebase gave a nice talk on this at RealtimeConf. ## 5. ~98% / 2% split for StreamData vs. SyncedData Most data does not need to be synchronized and instead is a stream. Streams are more compatible for offline and p2p architectures. Natural systems have no centralized synced state. Synchronization in Nature uses mechanisms like entrainment which uses an external media to propagate sync events between agents (eg. vibrations in the wall). Proper Natural Architectures will probably have 98% of our data designed as StreamData. Often, if we see the need for SyncedData, we probably haven't resolved the workflow around that data. StreamData Pub/Sub will be handled by WebRTC video and datachannels in order to scale and run offline. In the meantime, for cases where we feel we need SyncedData, we will use Firebase or a synching IndexedDB scheme for Synced data represented as VectorDatasets and RasterDatasets. Vectors will be GeoJSON in Firebase and Rasters will be Base64Datasets or RGBDatasets. ## 6. Models are deployed on layers ## 7. Store Geodata in Tiles Any data that has location will be stored in tiles. Tiles partition the world and are a Geographic Indexing System (alternative definition of GIS). Models will publish/subscribe to other models via tiles. Tiles can be either StreamData or SyncedData. ## 8. Data appears local Model code should have interfaces to read and write to data where location appears local. The background Acequia code should handle requesting and caching so the data appears that way. Note how Firebase pub/sub events that are sourced and sinked on the same VM happen locally before they are synced to a common server. ## 9. Defer signup The nexus of associations between cookies, local storage, QR codes, URL shares (via social, sms and email) and mobile phone numbers provide for most of the needs of authentication in a network of partially shared data. Request usernames and passwords as late as possible. ## 10. De-normal the new normal As our data storage moves away from relational DB into paradigms like NoSQL and not a database (noDB) structures our data is denormalized with the same data field in multiple locations for quick reads and minimal server-side query requirements. This goes against the grain of 20 years of data structures and we have to learn as a group new data structure strategies and patterns. ## 11. A pixel is not a little square and a voxel is not a little cube Pixels in our tiles and points in our point clouds are point samples (ie points with a surround sampling function), not little squares/cubes. > "This is not a religious issue. This is an issue that strikes right at the root of correct image (sprite) computing and the ability to correctly integrate (converge) the discrete and the continuous." > — Smith, Alvy Ray (1995) ## 12. Raster Tiles Similar to a definition of a function in mathematics where there is a distinct y-value for any x, our geo tilesets will have only one value among the tiles any xy (lat,long). ## 13. Quadtrees and octrees partition space to keep data volume constant ## 14. Sampling is hierarchy glue Allows models at different scales to publish/sub with each other and allows the transition from discrete/continuous and micro/macro relationships. Eg, a pheromone field of food-seekers in an ant system is a sample space of ant position on which the dual field of ants (nest-seekers act). Sampling is core in AgentScript as patches and turtles sample datasets (typed arrays). ## 15. PointClouds are stored in either 3D tiles or the fourth channel of RGBD for videos and images This is relatively uncharted territory and will require some experimentation. When to store as point cloud vs depth channels needs to be understood. Depth channels have a natural time component as well as the advantage of keeping the data under sousveillant control (eg, with the owner of the data). Cesium's 3D Tiles have potential. Potree manages this with octrees. Points will have RGB, XYZ and link to source image which will yield derived direction normal and sample size per pixel. Image data includes dateTime, owner, camera meta information (location, orientation, camera matrix, datetime, permission masks). - https://www.khronos.org/gltf - http://www.int-arch-photogramm-remote-sens-spatial-inf-sci.net/XL-3-W3/577/2015/isprsarchives-XL-3-W3-577-2015.pdf ## 16. Notes from Self (Directness, Uniformity, Modeless-ness) Our architecture is informed by the design philosophy of the language Self. Owen's colleague at Sun R&D Labs, Randy Smith, the author describes this philosophy here: https://www.youtube.com/watch?v=Ox5P7QyL774 - **Directness**: Go directly to an object to manipulate it. Inspecting a link, patch or turtle should allow the user to directly manipulate the attributes and code for that agent. At [time 18:33](https://youtu.be/Ox5P7QyL774?t=18m33s), Randy and Dave Ungar talk about this direct interaction. - **Uniformity**: All objects have basic uniform behavior. Similar to our base behaviors for agents, particles, patches and links. All objects automatically persist and are pub/sub'able. - **Modelessness**: Liveliness, Continuously running: no edit then run mode. Bret Victor live coding. This is partially what I mean as Github is where code goes to die. We should be able to edit an object in its context and keep it running. We shouldn't have a download, edit, deploy and relaunch from scratch. ## 17. Github is where code goes to die "Authoring is Always on" (frameworks are okay for now in github). Active Model code will be edited and served from the Acequia. Alan Kay likes to say this. Like Self, we should have direct interaction with the code on the page. ## 18. Mobile first ## 19. Offline first An offline-first strategy for Acequia ensures seamless functionality by prioritizing local-first design and peer-to-peer (P2P) data synchronization. Users can interact with the app and manage data autonomously, maintaining resilience during connectivity disruptions. When devices reconnect, Acequia synchronizes updates across peers, preserving consistency in a distributed system. ## 20. Apps are soft-assembled model views and controllers Many different apps will share channels - Channels are not unique to a single app. An app is the collection of model views and controllers. Controllers and Views interact with a model which can be direct data manipulation or through manipulation of collection of agents that then read/write to the data model. The data model persists in Acequia Channels which are implemented as Firebase URLs and WebRTC tiles. ## 21. Our System Architecture will be Nearly Decomposable A collection of channels will rarely be isolated to a single app but instead will be loosely coupled to other apps of a larger system (eg, a fire model will loosely couple to traffic, a weather model will loosely couple to a fire model, the larger wildfire app that includes many sub-models will loosely couple to the law enforcement system etc). This is consistent with Herb Simon's observation in "Architecture of Complexity" that complex systems are nearly decomposable. ## 22. Reed's law scaling via users, groups and roles ## 23. Models are the Unit Tests ## 24. Future: DataSync may migrate to WebRTC Our core data architecture heavily leverages Firebase for DataSync which is based on Websockets and is centralized. It is good enough for our current needs but is a scaffolding for what will eventually replace it. Firebase emerged when websockets came on to the stream and allowed for bidirectional realtime sync with the browser. As WebRTC becomes more standard, we will probably see a similar service emerge that leverages WebRTC and will be more distributed, and local than Firebase. As Firebase is satisfying our current needs, we won't distract ourselves by writing the WebRTC sync/persistence service from scratch in the near term - I expect there will be a good replacement in 2016.