Browsed by
Author: marcus

The First Rest API! (For the Hatch Baby Rest)

The First Rest API! (For the Hatch Baby Rest)

Ha ha, get it? The first Rest API? Programmer humor, right? Anyway, someone who knows that I love Smart Lights sent me a Hatch Baby Rest as a gift, and I was excited to unbox it and try it out. The Rest is a small nightlight that pairs with an Android or iOS app to set the color, auto-timer/schedule for turning on and off, and controlling the volume of the built-in speaker which plays soothing sounds.

Hatch Baby Rest Nightlight and App

Now, it’s supergreat that the light has an app, but I’ve got like 20 lights in my house and if each one had its own app I’d have a hard time keeping track of 20 different apps for controlling the lights. Luckily I’m a handy guy and I’ve got a home automation setup that could slot in this light quite nicely.

If only it had an API, that is. I mean, in this day and age, what smart light maker doesn’t have one?

clickity-clickity-click (sound of me writing to Hatch Baby to ask about it…)

Hatch Support’s pooh-pooh response

Hmmmm…. well that sucks…

So — you know where this is going, don’t you… fast forward a few hours and I’m happy to introduce to you a great .NET API for the Hatch Baby Rest. Thanks to Windows 10’s improved support (in the Creators Update 15063) for Bluetooth LE pair-less communication, it was actually pretty quick to do. UPDATE: I later on got a note from Ken (VP of software at Hatch Baby) that they’d be happy to contribute to OSS projects in the future, which is super awesome to hear coming from the company, kudos to them.

Here’s a sample, just a few lines of code. Hope this could be useful to other home automators.

GitHub Link: https://github.com/Marcus-L/m4rcus.HatchBaby.Rest, or install it from nuget:

Oh, p.s. it’s not a REST API, it’s a Rest API. I know, it was super confusing in the code too.

The Ultimate P2P Backup Software (or, the definition of insanity)

The Ultimate P2P Backup Software (or, the definition of insanity)

This week in “Things I Wish Existed”:

Free, easy to use, peer-to-peer data backup!

It’s a super good idea, right?

It would seem as though data backup would be a great business to be in–people are taking more digital photos and videos than ever before with the advent of great smartphone cameras, generating tons of data, and nowadays things that used to be physical, from mail to books, are more likely to be electronic–email and ebooks. More data. The data backup business could be worth $5.5 billion by 2023. But data backup is an unsexy business to be in, like selling seatbelts or toothbrushes or insurance–things which you probably need or probably should do or have, but maybe you don’t, at least as much as you should.

Peer-to-peer backup makes sense to me, as a software guy in an industry where lots of people have a computer or laptop (or two!), reasonably quick internet and definitely some extra storage capacity to spare, what with today’s multi-terabyte hard drives bringing the outright purchase cost of storage down to pennies per gigabyte. Heck, I’ve got three laptops, two headless servers, a few raspberry pi’s (and a partridge in a pear tree) with excess capacity running at home right now! I’ve got more than enough devices to back them all up to each other, and could trade space with friends in Boston or family in Honolulu to ensure that my data would be protected from local disasters or theft.

The idea is not new, but it seems as though the internet is littered with the debris of companies and projects trying to get some kind of P2P backup service/software to catch on that “regular people” can use.

Now I explain that everyone who has tried this has failed

Definition of Insanity: doing the same thing over and over again and expecting different results

— originally said by: “no one really knows

CrashPlan shutdown notice

It was a great run for CrashPlan for Home, launched in 2007 and helping me personally back up hundreds of gigabytes of data to and from computers around the world. But apparently cheapskates like me (who use the software for P2P backup instead of paying for cloud backup services), and data-hoarders who accumulate terabytes of data and back it up to the all-you-can-eat flat-rate service make consumer backup a relatively unprofitable endeavor, so they just announced they’re shutting it down.

While CrashPlan was not the only P2P backup system, it probably was one of the most widely used due to its relative ease of use, low cost ($0 if you only back up P2P) and ubiquity (runs on Windows, macOS and Linux). For sure there were things about it which were not so good–the backup client was ugly, slow, and somewhat complicated. But at least it was P2P backup and it worked. So with the news of the P2P service’s demise, much hand wringing and soul searching has commenced. People are jumping out of the sinking CrashPlan ship and looking for another horse to hitch their data wagon to.

I guess it never really meshed

 

What’s a Wuala
It’s because you called it Cubby

There apparently was a service called Wuala which did P2P backup until 2015, there was something else called BuddyBackup which hasn’t been updated since 2014, and some guy wrote about P2P backup at BackupCoop.com in 2006, and there’s an abandoned project Backup P2P still downloadable, and a P2P backup utility DataHaven.NET which the author sunsetted to start a P2P backup company called BitDust (insert joke about what his company is eventually going to do here)…

So many failed companies and abandoned products! Does this serve as evidence that A) there is no market for it, or B) the cost and hassle of setting up backup is still too high, or that C) all attempts to date at cracking this problem have been substandard? What would a backup product look like which might have a chance of succeeding where so many attempts have not?

The part where I dream a little and oversimplify a lot

This is getting a little long but my first reaction as a software developer was to start imagining the wonderful future where something awesome exists, and how hard it would be to implement this, so let me just brain-dump some ideas for roughly what you’d need to do and then drop the mic:

The Ultimate P2P backup system:

  • Everyone should use it! Therefore it must be easy to use, and have incentives to be used.
  • It should be super cheap! So it should be P2P to utilize cheap excess capacity.
  • No one should be able to hack it! Accordingly it should be end-to-end encrypted.
  • Not all data needs hardcore backup! Ergo the level redundancy should match the importance of the data.

Here are some developer-speak ideas for making this thing a reality.

Step 1: Create backup client

Like this one, but not super ugly, slow, and hard to use
  • Cannot stress enough the importance of ease-of-use in the discovery, installation and usage of the client UI being paramount to widespread adoption, make it beautiful, simple and powerful.
  • Intelligent consumer/user defaults – backup user profile directories, exclude temp/swap/transient files. Make it easy to choose what to back up and how many copies to make.
  • Tech stack – do the client in something like Electron so you can reuse most UI code across platforms

Step 2: Implement Service

  • The backup service code needs to be fast but portable since it’s doing a lot of stuff, something cross platform and productive like .NET Core or Go.
  • Data Deduplication is super important, need to either do it file- or block-level – crib something from borg or sdfs or btrfs.
  • Map files to file versions to blocks to block hashes in a database (mariadb?) – make keeping version history as cheap as possible (only need to store changed blocks for multiple versions). Also need to have a way of pruning old versions periodically (along with unused blocks).
  • File/Directory level redundancy specification, like Storage Spaces failure tolerance levels. Limited by storage quota and number of peers.
  • Should accommodate re-distribution/leveling for adding and removing peers/quota (whether intentionally or unintentionally)
  • Ought to be able to run headless on Smart NAS devices like Synology or QNAP or IoT devices like Raspberry Pi, so ARM support for runtime is important

Step 3: Create P2P network

  • Reuse something like Satoshi Client Node Discovery or PEX to find peers
  • Gotta bust through firewalls/NATs with UPnP or NAT-PMP
  • Allow users to proffer storage quotas to friends, or trade storage with strangers. Assumption is that friends are reliable/stable storage targets but strangers less so
  • Store file/version/blockhash database locally for fast access, need to also replicate encrypted copies to peers for recovery in case of computer crashes
  • Peers receive encrypted blocks/db-replicas, not able to read them without the key
  • Controls for throttling bandwidth based on network/time/cpu conditions

There’s the master plan, will you please go make it?

It’ll be really hard (the deduplication and quota/redundancy leveling problems sound super extra hard, whereas making a passable P2P discovery/networking and working with various OS filesystems is only normal very hard). You might have to run and pay for a few servers to help with peer discovery. You’d definitely want to open source it to get the crypto nerd early adopters to trust it. That would also help with continuity and maintenance if you could recruit enough developers with a vested interest in seeing it succeed. And in the end you’ll be paid in zero dollars but also in the currency of satisfaction in that you have helped solve one of the computing world’s biggest problems. The data-safety equivalent of helping us all get a little insurance, brush our teeth and buckle our safety belts.

Addendum: Additional ideas to maybe make money (or “Step 4: Profit?”)

  • Build the P2P backup software into a branded NAS or partner with a hard drive company to make drop-in self contained peers, sell them like hotcakes

  • Or maybe put some hard drives into a Smart Home device like Google Home or Echo Dot and make those drop in peers. Or do like Apple does it for their Airport Time Capsule (hard drive backup in a WiFi router), only this time with P2P backup they’d be better utilized and internet connected. (Sell them like hotcakes)

  • Start tons of companies to sell backup space! With the P2P backup network, anyone could create a CrashPlan/BackBlaze/Carbonite backup service with low overhead by simply firing up the software on their servers and offering to “peer” storage quota to you for a price. If lots of companies started up to do this, they could compete the price of offering quota down to reasonable levels. And they could differentiate themselves by economics (making it dirt cheap by any means necessary) or security (ultra-secure datacenters deep underground) or geography (servers in politically insulated countries with strict data extradition policies), while all using a common network and software platform

  • Or maybe you already have a company with a datacenter with excess capacity! Rent the full storage capacity out immediately to help recover the sunk cost, while dialing back quotas when you need the storage space back (and letting the P2P network figure out re-leveling the redundancy for those affected)

  • Get Acquired! Maybe you decide to do this as a VC funded startup or build a prototype and then have Microsoft or Apple or Google buy you up to build the technology directly into their operatin systems. That would not be super great since then the P2P backup software wouldn’t be open and free any more, but I guess you could cry about that all the way to the bank.

Mic Drop

OK, that was long enough. I really hope someone goes ahead and builds something like this. If no one else builds it by the time CrashPlan Home stops working in October 2018, maybe I will. Just because everyone else has failed, doesn’t mean I would too, does it? Or was that the definition of insanity?

Wake up with the Sunrise

Wake up with the Sunrise

I’ve been trying to get up earlier in the morning, so rather than just going to bed earlier, I decided that I needed a wake up light that gradually illuminates like a sunrise, while playing soothing wake up sounds like birds chirping.

My Wake-Up-System wishlist is:

  • The wake up system should fade the lights in slowly like a sunrise
  • It should play wake up sounds (birds chirping) softly at first then progressively louder
  • There should be no annoying glowing lights on the device(s) to keep you up
  • The alarm time should be changeable without using stupid tiny digital alarm clock buttons
  • Both bed occupants can change the wake time easily from our phones
  • Both bed occupants can silence the wake sounds easily from phone or a remote control
The Philips HF3550, a very expensive object

I looked on Amazon and something like this exists, but the damn thing is $200! Also it has pretty terrible reviews, isn’t compatible with Android1, and being just one light it’d end up not being bright enough to wake both occupants of the bed without being blinding (or out of view) for the other. What to do? Buy two? That would be expensive, and the device itself didn’t meet a few wishlist bullets.

So once again instead of deciding to just go to bed earlier, I remembered that I had a pair of Hue Bloom light bulbs packed away in a box. What if I could use those to simulate a sunrise? I set them up and installed the Hue app, which had a built-in scheduled sunrise wake up function. However, there didn’t seem to be a way to play sounds with the lights fading in. I could probably find an app for that, but then there’d be two different things to change around or enable/disable when the wake up schedule is changed, which would have a pretty unacceptable WAF.

I came up empty in a search for such a device, so after a day of tinkering around, I am excited to introduce to you: Sunrise!

The alarm is set from the app, from any phone (it’s a web app with “add-to homescreen” features) and runs off a Raspberry Pi in my closet. Best of all I was able to hook it up to the Hue lights I already own, play wake up music on the Chromecast Audio which was already hooked up to speakers in the room, and works with the X10 remotes I was already using for some other lights in the room.

It works surprisingly great for the duct-taped-togetherness of it all. It’s so nice to have the lights gradually turn on and wake up to the sound of waves crashing or birds chirping, rather than being jolted out of bed with the usual smartphone alarm. I highly recommend it. We’ve each got our own remotes on the side of the bed so both can silence the alarm, turn on the lights and reschedule the alarm without getting out of bed! Bliss.

If you’re sufficiently nerdy (or motivated), feel free to replicate my setup using my code as a starting point, it’s up on GitHub. Happy sleeping (and waking)!

 

  1. A total non-starter, although there’s also a non-iPhone version but the reviews on that say that either the buttons were terrible or the clock LEDs weren’t easily dimmable