Behind the scenes on Invisible Roommates, our new work for SPACE10 & IKEA
and a little bit about the videogames i'm making
Can’t believe no one told me that when you create a newsletter, one day you actually have to write in your newsletter, but here we are - this is the ✨inaugural post✨ in my newsletter where I said I would infrequently share new work.
I’ll talk a little bit about this new project that was just released, and update you on some of the other long term stuff I’ve been working on. So here’s Invisible Roommates.
Invisible Roommates
Invisible Roommates is an AR app that reveals the secret lives of the devices you share your home with.
This project was a collaboration with my friend, the inimitable Eran Hilleli, and was made for SPACE10 and IKEA (don’t feel like I need to link their website lol) as part of a new collection of their Everyday Experiments under the theme “Privacy & Trust”.
You can see more about it on the project page, and also in a Twitter thread I did about it, but I thought I would talk a little bit more about the behind the scenes here, and share some of what the early prototyping looked like.
When SPACE10 approached us with the brief to make a project exploring the theme of Privacy & Trust in the home, both Eran and I were a bit like 😰 because it’s a serious topic that required a level of thought and care in order to do it justice. Both of us also tend to make work that leans playful (our previous collaboration was this project called Garden Friends), so it took some time and brainstorming to come up with an idea for something that felt true to us but also addressed the theme.
For me, it was important to create something that was both 1) real, not speculative in its technology and 2) value-neutral, not Black Mirrorly dystopian or Elon Muskly utopian. We had a few ideas, and ended up landing on this one:
It felt right to explore the concept of making invisible things visible, but in a way that was particularly playful and accessible to a non-technical audience. So Eran began working on the characters, and I started to figure out the tech stack.
There were two main components to the app - a node server running on the computer, and a Unity app running on the phone. The node server was used to do the device and packet detection, and the Unity app is where all the AR happened. I used websockets to connect the two. I’ll talk about the details of each part, starting with the node server.
Device and packet detection
There are a few different ways to detect devices around you - Bluetooth, mDNS, ARP - and after exploring some different options, it felt important to also be able to capture the network traffic between devices. After all, it wasn’t just about what devices there are in your space, but also about being able to visualize some level of what is going on between them. So where I landed was using a node library called local-devices, which is pretty much a wrapper for arp -a, allowing me to find the IP addresses and MAC addresses for all of the devices on the same network as my computer.
A MAC address is basically a unique ID for your device (it has nothing to do with Apple) - it’s a string that each device has, which looks something like c1:49:ae:0b:16:97. With the MAC addresses of my devices found, I used an API called macaddress.io to programmatically look up the manufacturing info of each device.
The next piece was to intercept and log the traffic between my devices over the network. This is hilariously called “packet sniffing.” I used a node pcap library for this. Packet sniffing allows you to see which devices are communicating with each other, though it’s not easy to find out the exact contents of the packets themselves:
(By the way, you should read this piece by Surya Mattu and Kashmir Hill about data trails and packet sniffing.)
From all this, my script could now see, in real time, which devices in my house were sending data to each other.
Putting the characters in AR
In the meantime, Eran had been developing these adorable characters that looked like little house spirit versions of the devices they would represent. We put them into Unity, and used ARFoundation to handle the AR piece. Eran added animations for waking, sleeping, walking, talking, dancing, and shooting paper airplanes.
I built the logic to generate the correct characters and animate them accordingly, and so what was left was hooking up the node server to the Unity app so it could send data in real time.
Connecting the server and the app
Since my node script was now detecting packets, and the Unity scene was set up, the part that was left was being able to send data from the node script to the app on the phone while it was running.
For this, I used my longtime friend socket.io, and this Unity package socket.io-unity to allow my Unity app to be a client for my socket server. To be honest, this part was a bit of a mess - there were all sorts of issues with threading in Unity that made this tricky, but in the end it worked, and so my node server could tell Unity in realtime what was happening with the devices on the network.
Putting it all together
So now the pieces were all there, and it became about refining the animations, some of the AR stuff (we had some trouble with the shadow planes and occlusion as you can see above) and the look of everything.
But it was great to see it working - I could see that there was always a lot of action from my computer and my router (makes sense), and I think the little paper airplanes traveling around added a lot of flavor to the scene.
It was also cool to see, for example, that when I had the computer and printer characters in the scene, and I sent a print job from my computer, the printer would awaken from it’s “sleeping” state, the computer would send it paper airplanes, and my actual printer would print something out.
Cool but where can I download this
People have asked us if they can download this app, and the answer is: sorry, no. The reason for that is because the scope of this work was for a prototype, and it would take (a lot) more to make and support something that could be releasable. (I’m sure if you are also a developer you’d understand 🥺)
Obviously, the two-part local node server + app thing wouldn’t work on a larger scale, so that would be one thing that would have to change to make this into a releasable app.
Final thoughts
For me, a lot of this project hearkened back a class I took in grad school a few years ago: Tom Igoe’s Understanding Networks (so shoutout to Tom).
I’m happy with what we did here - the technology we used isn’t groundbreaking by any means, but as I said above, it was important to me that this project was “real,” and not speculative renders. Technology is ~happening~ all around us, and I think there is value in making the invisible things visible, as well as in taking a more playful approach to visualizing something as dense and inscrutable as packet sniffing. Also I’m just obsessed with the characters Eran created, and it was so delightful to work with him again. Thanks again to SPACE10 and IKEA for supporting Invisible Roommates.
Videogames?????
Okay, so I intended to write a bit about other things I’m doing, but this ended up being so long that I think I’ll just save it for next time.
As a basic overview, I’ve spent the vast majority of the last couple years(!) working on two very different videogames, let’s say Game A and Game B, both of which aren’t really ready for me to talk about publicly yet. As you may know, I left my creative technologist gig at Google in 2019 to become an independent game developer (and I have a lot to say about that which I’ll save for later), and if there’s one thing I’ve learned is that it takes a fucking long time to make videogames 🥺 🥺 🥺.
But, these games are by far the most ambitious, biggest projects I’ve ever worked on (with many amazing collaborators), and I hope I can talk about them more soon!! I think I’ll write my next post about an amazing and very unusual method of playtesting we did for Game A, so stay tuned for that.
Well, thanks for reading my first newsletter post. I’d love to hear your thoughts or questions - by the way, I’ve been having an incredible time streaming on Twitch over the last year, so feel free to come by sometime. OKAY BYE
Great post ☺️
while i understand a formal release would be hard, if u ever felt like uploading the source code, i think that'd be really really cool
Good