Project #2: My Location
Pushing from Mobile
November 7, 2016
Introduction
This post continues a previous entry. But bringing everyone up to speed—
I am traveling, and want the About page of this website to programatically say where I'm living. The plan is to post my location every hour to a remote database. When someone visits the page, a client-side GET request is made to the database and JS will fill in the City and State on the page.
The challenging part is posting my location.
I couldn't easily figure out how to do this from my phone,
so I set it up to run from my laptop
(using a Ruby script called with cron).
It worked, but there were two major drawbacks. 1. My computer is not always on. 2. My computer is not always with me. Therefore, my location wasn't very accurate.
I put this project on hold for a few months. But I recently revisited it and now having it working using my phone's location!
Mobile to Database (Direct): Failed Attempt
A friend1 told me that IFTTT (mobile app) can run web requests on timed intervals. When I heard this, I thought I'd have my location posting from my phone within an hour.
Unfortunately, IFTTT is not very robust2. It doesn't allow the chaining of "recipes" and I couldn't figure out a way to calculate a parameter and pass it with a request.
I wanted to directly make a POST request to the database (Firebase). But there was no way to determine the city and state or longitude and latitude and include that with the request3.
Serverless: Failed Attempt
As a result, I thought I could keep the IFTTT request and get the mobile location from the IP address of the request.
As a way to think about it, I'd make a simple request to a remote endpoint every hour. The application behind the endpoint would run some logic to determine the physical address from the IP. Then it would post that information to the database.
But instead of an application, I'd use Serverless Architecture!
Serverless is all the rage these days, and I'd been looking for an excuse to play with it. Serverless is also known as 'Backend as a Service' and 'Function as a Service' and 'NoOps'4. The best explanation I've read is available on Martin Fowler's website.
This seemed like the perfect use-case for Serverless. The idea is—it's wasteful to run an entire application to do one thing. AWS offers a service Lambda.
The way Lambda works is that whenever it is "triggered" it executes a function. The advantages are—
- you don't need to run a server
- because there is no server, you don't need to worry about operations or infrastructure
- it's much cheaper. instead of paying for an instance 24/7, you only pay when functions are run
I won't go deep into the details, but I set up an API Gateway for the trigger event. Whenever the endpoint is hit, it runs the function on Lambda. The code for the function is available in a Gist 5.
This function does three things. It gathers the important information: datetime and IP address. It transforms the IP address into a city and state. It posts that information to the remote database.
Believe it or not, this all worked. I was feeling so pleased with myself, I created a diagram.
Unfortunately, there is one major problem. It is not possible to determine a physical location from an IP address when connected to the wireless mobile internet.
This works end-to-end with the phone connected to WiFi. But when I was connected to the mobile web, my IP address always had me in Virginia 😃 6.
Web Scraping: Success
I decided, fundamentally, that trying to transform an IP into a location was a mistake. I needed to get the physical location from the phone itself.
I did some google research for "How to share my location with an Android phone". Sure enough, you can just change certain privacy settings and share your exact location with the world on Google+.
Once I was sharing my location on Google+, I just needed to: scrape the page every hour, grab the location, and post it to the remote DB.
For visiting the page, I used Watir because it looked popular and let me write Ruby. The code is available in a Gist.
Watir runs a real web browser, so it's easy to click and type and sleep as necessay 7.
I got this working quickly on my local computer, but there were a few gotchas when moving the script to a server. 1. Watir needs to run headless, because a server cannot open GUI Chrome. 2. You can use any web browser, but you need to install it and the driver 8.
What Did We Learn?
- Test Assumptions Early
- It would have only taken a minute to walk outside, get my IP address on mobile internet, and seen if that translated to a correct physical location.
- This would have saved me hours working through the entire first approaches.
- But learning about Lambda was the most fun part of this project, so I don't feel too bad.
- Your Users Don't Care About Your Tech Stack
- I'm sad the Serverless approach didn't work. Because, let's face it, that approach is cool and web scraping is boring.
- That is, after all this—it's disappointing that my solution is just using Google+ and grabbing the info from there every hour.
- But it's a good reminder that your users don't care. My mom can visit the page and see where I am, and how that happens truly doesn't matter to her.
- Build Robust Software
- On the flip side of that, it is important to build robust software.
- If Google+ renames the link from 'About' to 'Details' my script will break. (I hard-coded "click the link About".)
- Even through your code doesn't matter to end-uses, it still needs to hold up.
- It's Okay To Put Things On The Shelf
- I spend a lot of time feeling bad about all the things I'm not getting done.
- It's okay to pause on a project. Sometimes distance brings clarity, insight, perspective, and/or motivation.
- Thought of Serverless/Lambda
- I was really impressed using Lambda. I now understand all the hype. I'm confident I'll be using it again soon on one project or another.
- The tools are more underdeveloped than I expected. For example, the only supported languages are: Node, Python, and Java.
- When it's more mature in a few years, it will be all-around boss.
Conclusion
I'm happy to have this working. My location really does make it from my phone to my About page. When I drove from Memphis to Austin, I kept checking and it was fun to see that I lived in "Marion, Arkansas", "Gurdon, Arkansas", "Texarkana, Arkansas", "Lancaster, Texas", throughout the day.
There is one last thing I don't want to gloss over: privacy. My solution only works because I am publicly sharing my exact location. This doesn't bother me; in fact, I like it9. But I highlight it to say that this definitely doesn't work for everyone and there is a sacrifice/cost to the approach I used.
If I were to do it again, I think building a native mobile-app is the right approach. There's no way getting the location of the phone should be so difficult. Even though I know nothing about mobile-dev, getting the location and making a POST request sound pretty simple. I'm confident I could struggle through it with some copy-pasta of tutorial code.
Perhaps I'll give it a stab in a few months.
[UPDATE: I moved to Seattle, so hard-coded the location on the About page and turned off the script. I'll need to get a more robust solution in place before my next big adventure :) ]
1 shoutout to Erik Olsen.
2 IFTTT, which stands for "if this, then that" is incredibly cool. However, it is made more for general users rather than developers or power users.
3 I had the idea to create a recipe to write the City and State to an Evernote list at, say 10 minutes after the hour. And then grab the details from Evernote to pass to the webrequest at, say 15 minutes after the hour. But 1. I didn't think I could get it to work and 2. that feels incredibly hacky. I mention it only to give an idea of what IFTTT makes possible.
4 In my honest option, Serverless is a terrible name. For what I'm doing, 'Function as a Service' seems the most accurate. But, damn, it's fun saying Serverless.
5 This code is not good. If you are laughing at my date parsing, laugh away. Because this approach didn't pan out, I never refactored. And I'm terrible at writing JavaScript. But I wanted to provide an idea of what a function actually looks like.
6 Intuitively, this makes sense. I should have thought about both scenarios earlier. I started reading about mobile IP addresses, but didn't have the patience to learn exactly what's going on.
7 It seems ideal for setting up your daily entry to the Hamilton lottery, for example.
8 Instructions to install server-chrome available here. Chrome driver available here. You'll need to
wget and chmod +x.
9 I believe strongly in transparency, and keep toying with the idea of doing something crazy like making my emails open.
Have anything to say? Questions or feedback? Tweet at me @cmmn_nighthawk!