keshav.is / building / automated lights

Automated lighting

This project write-up is a work in progress, as I need to compile a bunch of code, pictures, and videos.

In my San Francisco apartment, I automated my Noon light switches to automatically turn my lights on and off as I walked around. Technology has progressed by leaps and bounds since my Jarvis project, so I was able to assemble all of this with Raspberry Pi, RPi cameras, simple mounting hardware, and TensorFlow object recognition.

Proof of concept

I started by mounting two cameras, one facing the kitchen and another facing the living/dining area. I initially set up a simple Python script for motion detection which would hit the Noon API, and turn on the lights as I moved around, and turn off the lights when no motion is detected.

Noon API

Noon Home does not have a documented API, and they gave an indefinite ETA on when one would be available. So I fired up Charles Proxy and observed the API requests from their iOS app.

I was able to collect the UUIDs associated with each of my light switches, and make simple POST requests to the Noon API to switch the lights on and off. I had 8 Noon switches installed, along with lamps, Phillips Hue LED bulbs, and several smart plugs. The API for connecting with these devices was easy to figure out through GitHub - link todo

For the only device that doesn't Star Wars references in API endpoints (https://bobbajo.api.noonhome.com)

{
    "space": "5cd66ef0-5760-4008-80b6-e658bafbf811",
    "activeScene": "5cd66faa-b4d8-40bd-8083-4301758a21bb",
    "on": true,
    "tid": 6703
}

Object detection

I then set up a Mac mini with TensorFlow running the off-the-shelf object recognition model. It was extremely accurate right out of the gate at recognizing me in the frame.

I then mapped the center and size of each bounding rectangle into a neural network with 4 inputs and one output for each light. When a particular output value exceeded a threshold, the light would turn on. I trained the network by standing in various places in my apartment and manually turning lights on and off to create a training data set. Even though the resulting network worked well, for the next couple weeks, I had to get used to lights randomly turning on/off or not turning on when I wanted them to.

It took some time to build a good system for storing training data and storing error corrections - when the network doesn't respond as expected and I manually change a light switch, I wanted the exception to change the network without my manual intervention. This was the point where the complexity of caching frames, getting WebSocket updates from the Noon API, and trying to build a custom app for the system pushed this project to the back burner.