Proper Counter Cache Migrations in Rails Feb 5, 2016

This post is less about the basics of counter_cache support in Rails and more about best practices for introducing a new counter cache to an existing project. More specifically, the goal is to detail the most efficient way way to create an ActiveRecord migration to support a new counter cache column.

The Problem

You see, the interweb is currently filled with some poor advice about how to seed a counter cache column.

Now that you’ve clicked on those examples, please erase them from your memory. Iterating over the entire table, loading each record into ruby-space (without batching, mind you), and relying on either update_counters or reset_counters in your migration is a sure way for your next deployment to take minutes to finish. It doesn’t take millions of records to get hit with this pain either.

The Example

Let’s assume that we have some stereotypical tables named posts and comments. Let’s also assume that we decided to add a comments_count counter cache column to the posts table.

The Migration

Given this example, your migration should look something like this:

class AddCommentsCountToPosts < ActiveRecord::Migration
  def change
    change_table :posts do |t|
      t.integer :comments_count, default: 0
    end

    reversible do |dir|
      dir.up { data }
    end
  end

  def data
    execute <<-SQL.squish
        UPDATE posts
           SET comments_count = (SELECT count(1)
                                   FROM comments
                                  WHERE comments.post_id = posts.id)
    SQL
  end
end

The Results

With over to 25,000 posts and 100,000 comments, using SQL will take seconds instead of minutes.

-- execute("UPDATE posts SET comments_count = (SELECT count(1) FROM comments WHERE comments.post_id = posts.id)")
   -> 1.3197s
   -> 26900 rows

Let’s compare that with how long it would have taken if we used a Post.reset_counters approach:

-- Seeding posts.comments_count -- Better grab a coffee.
   -> ...........................
   -> 144.7302s
   -> 26900 rows

Here is the actual code used to run these two examples.

The Moral

Sometimes SQL can be straightforward and fun.

Reverse Engineering an American Express USB WebKey May 26, 2015

I received a strange piece of hardware in the mail the other day. American Express is piggy-backing on LegalZoom document deliveries with an interesting piece of marketing – a USB key-like device in the shape of a credit card.

American Express Fake USB Keyboard

Thinking it was a USB drive and one that I might be able to format and reuse for something else, I plugged it in to my Mac. To my surprise, it immediately opened Safari and a website pointing to an American Express credit card offer. Huh? I thought OS X didn’t have Autorun support. So, of course, I tried it again. This time, I noticed that the address bar in Safari scrolled as if someone was typing really fast.

Holy trickery, Batman. It’s mimicking a USB keyboard!

Here are the steps that it takes:

  • ⌃F3 to focus the Dock
  • Types: Safari
  • return to launch Safari
  • ⌘L to focus the address bar
  • Types: http://www262.americanexpress.com/landing-page/business-cards/sclp/bgold/aff0022/44460?PID=15&BUID=SBS&PSKU=BGR&CRTV=LZBGRLTOWBKYREF15
  • return to launch site

NOTE: If Safari isn’t running or pinned to your Dock, the entire process fails. They probably would have been better served to start with ⌘-space and a Spotlight search instead.

Digging In

My curiosity was piqued, and this reminded me of the USBdriveby project that Samy Kamkar discussed on a recent Tim Ferriss podcast.

I wondered if I could reprogram this American Express USB device to behave in a similar way as the Teensy USB Microcontroller that Samy used.

Amex usb key
"AmEx USB Stick" snapped from it's marketing enclosure.
Teensy
Teensy 3.1

Find the USB Device

First, let’s see what OS X thinks about this USB device. We’ll do that by inspecting all USB devices both before and after plugging in the USB key. The difference will help us find the device itself.

system_profiler SPUSBDataType > ~/usbout.txt
system_profiler SPUSBDataType > ~/usbin.txt
diff ~/usbout.txt ~/usbin.txt

This should yield a result like the following:

33a34,44
>         WEBKEY:
>
>           Product ID: 0x6662
>           Vendor ID: 0x05ac  (Apple Inc.)
>           Version: 8.15
>           Speed: Up to 12 Mb/sec
>           Manufacturer: TP6662
>           Location ID: 0x14500000 / 27
>           Current Available (mA): 500
>           Current Required (mA): 100

Note the Product ID and Vendor ID values. We’ll need those later.

I find it fairly suspicious that it mimics hardware from “Apple Inc.” Kinda sketchy.

Now What?

At this point, I was immediately stuck, but after some research, I found a superuser question on Stack Exchange titled Can I “reprogram” an American Express USB drive? No direct answers, but this answer gave me hope:

It consists of a Cypress PSoC controller, combined with a 24c02 serial eeprom … With a simple program called PonyProg you can read and modify the contents of the eeprom. http://superuser.com/a/216810/8424

I spun my wheels for a long while with no luck, but later, I found an article titled Hacking USB Webkeys. This looked promising.

Hardware Surgery (and Failure)

I slowly pried the chip from it’s plastic enclosure. It was attached with some glue.

Amex usb chip
"AmEx USB Chip" torn from it's plastic enclosure.

Sweet. Contact points that I should be able to use to access the EEPROM.

All seemed well, until I tested the webkey again. I must have damaged something, because it now skips every 5th to 8th character as it emulates a keyboard. Shoot. Maybe next time.

Let’s [Pretend to] Read all the EEPROM

Disclaimer: I wasn’t able to test any of this, because I damaged my hardware, but I encourage others to carry on where I left off and let me know if you have any success!

As it turns out, PonyProg (the program mentioned on the Stack Exchange post) doesn’t work on OS X, but I found a program called ch341eeprom that claimed to perform a similar task.

It depends on the WinChipHead CH341A.

Winchiphead ch341a ic
WinChipHead CH341A

It also depends on libusb, so let’s install that using Homebrew and then clone the repository so we can compile ch341eeprom.

brew install libusb
git clone https://github.com/commandtab/ch341eeprom.git
cd ch341eeprom

Compile, and try to read the eeprom:

make
ch341eeprom -v -s 24c02 -r ~/eeprom.bin

Conclusion

While my attempt at hacking this hardware resulted in catastrophic failure, don’t let that stop you from trying something new or breaking down an unknown piece of tech. I learned a lot during this little investigative project, and if I find another free marketing webkey in the mail, I just might pick-up where I left off.

Rejuvenating My Piece of the Interwebs May 22, 2015

This is something that I’ve wanted to do for a long time, and I was finally able to get my site redesigned and relaunched this week. The old design made it 5 years.

New, Responsive, and Light:

New Responsive Blog

Old, Rigid, and Dark:

Old Unresponsive Blog

The old design served me well, but it had a few problems. For one, it didn’t work well on small mobile browsers. Now, it’s much more responsive to all browser sizes. I also used to get complaints about my big headshot in the bottom right corner. While that self-promotion-marketing-hack helped people recognize me at conferences and events, I think it freaked out some of the more scopophobic readers amongst us. My headshot-hack remains, but I now hide away when you start scrolling. For what it’s worth, my 3-year-old daughter thinks it’s fun to play peek-a-boo with it, so there’s that.

I think the new site design still needs some extra work, but it’s good enough to launch and start gathering feedback for now.

What Changed?

This was basically a total rewrite. A lot has changed, but here are some of the key updates:

  • Responsiveness: Better views, font-sizes, and readability across all browser sizes.
  • No More Comments: Yup, I dropped all my disqus comments. This might upset some, but it is going to make my life just a little bit better. Instead, I’ve enabled Twitter’s Web Intents at the bottom of each page. You can now reply to me via Twitter. For old posts, I did my best to find my original tweet about the post. For new posts, I’ll do my best to keep everything synchronized.
  • Jekyll to Middleman: The original site was statically generated by Jekyll. Jekyll 2 made a lot of great improvements, and I’m sure Jekyll 3 will continue that trend, but Middleman has just always felt so much cleaner to me; it’s organized in a way that I like to work. Plus, it’s easier for me to standardize on one static site generator across the sites that I maintain, and I honed in on Middleman well before Jekyll v2 launched.
  • Twitter Card and Open Graph Support: You know those nice catchy pictures you get when you share a website on social media? Yeah, I got that. Learn more about Twitter Cards and Open Graph.

What’s Next?

I have a lot of little plans for this site, including doing a better job of promoting the things that I work on day-to-day, but in general, I hope the redesign also rejuvenates my desire to publish new relevant content about programming, the web, business, entrepreneurship, tech, etc. No promises, but that’s the goal.

Let me know what you think. Enjoy!

Talk: Cloud Services You Should Be Using To Build Your Startup May 12, 2015

BOULDER, CO — Boulder Ruby — I had the opportunity to present at Boulder Ruby during Boulder Startup Week 2015. I covered several online services that not only have excellent Ruby support, but also can help accelerate your MVP and/or product launch. The finale included a live demo of Cloudinary, Pusher, and Embed.ly services.

HTML5 Autocomplete Cheatsheet Aug 20, 2014

I was recently frustrated with the documentation on the HTML5 autocomplete attributes and Chrome’s response with the x-autocompletetype vendor specific attribute.

These attributes are typically used on <input> elements to better signal to the browser the best way to “autofill” a form. The problem, however, is that there is no official standard yet, and there are mild differences between values used for autocomplete and values used for x-autocompletetype.

The best documentation I could find on autocomplete types was on whatwg.org, but the formatting made it a bit difficult to easily see what types were available/recommended.

Hence, this cheatsheet was born. Let me know if it helps.

Autocomplete Types Cheatsheet

Continuous Integration and Deployment with Middleman, Codeship, and GitHub Pages Dec 19, 2013

I use GitHub Pages for hosting some of my websites, and I use the Middleman static site generator as my content management system for some of these sites.

I like to run continuous integration for my projects whenever possible, and this goes for static site repositories as much as regular code repositories.

Recently, I started playing with Codeship. It’s a well-priced continuous integration and deployment service, and I wanted to be able to automatically deploy my middleman site upon successful builds to the master branch. It took a bit of trial and error, but I finally got something that works well.

Test Settings

After initializing a new GitHub repository in Codeship, you’ll need to define how tests are run. Under the Test project settings in Codeship, select Ruby as the language and define your Setup Commands and Test Commands.

Setup Commands:

bundle install

Test Commands:

bundle exec middleman build

Perfect. Now your site will run a test build upon every code push.

Deployment Settings

This is where I had some trouble, but I finally found a set of commands that correctly deloyed to our gh-pages branch in the repository. First, it’s worth mentioning that I use the middleman-gh-pages gem for deploying to the gh-pages branch. This gem gives you a rake publish task that handles most of the dirty work.

Under the Deployment project settings in Codeship, configure a deployment from the master branch using a Custom Script.

Custom Script:

git config --global user.email "robot@example.com"
git config --global user.name "Codeship Robot"
rm -rf build
git remote set-branches --add origin gh-pages
git fetch
bundle exec rake publish
sleep 30
wget --retry-connrefused --no-check-certificate -T 60 http://yoursite.com/

This configures a git user, removes the build directory that was left there from the test steps, adds a gh-pages remote branch (because Codeship only clones the relevant branch during setup), and runs the rake publish task to deploy the site to the gh-pages branch.

If all went well, your site will automatically deploy to GitHub Pages upon a push and successful build in the master branch.

GitHub Publishing Caveat

Make sure the email address that you use for this git user is a verified email address in your GitHub account. Otherwise, GitHub will accept the commit, but it will not publish the changes to your site. To fully make everything work, you will need to do one of two things: 1) Move the Codeship deploy key to an SSH key in your own GitHub account OR 2) Create a new machine user on GitHub and move the Codeship deploy key to an SSH key on that user account, give that machine user push/pull access to the repository, and make sure that machine user’s email address is fully verified by GitHub. I highly recommend going with option #2.

Extra Credit: Force Codeship to skip builds on the gh-pages branch.

Unfortunately, Codeship currently tries to run builds on all branches, including the gh-pages branch. This is undesirable for this setup, so to avoid this, we also need to add “–skip-ci” or “[skip ci]” to the commit message that is pushed to the gh-pages branch.

Fortunately, after this pull-request by yours truly, middleman-gh-pages can support that.

If using version >= 0.0.3 (or the master branch), you can add this to the bottom of your project’s Rakefile:

# Ensure builds are skipped when pushing to the gh-pages branch
ENV["COMMIT_MESSAGE_SUFFIX"] = "[skip ci]"

TEDx Talk: What makes someone Successful? Dec 5, 2013

LEESBERG, VA — TEDxBalchDriveWomen 2013 — Software — “Luck” is too often attributed to one’s success. Let’s cut the modesty and tell better stories about the good and bad decisions that got us where we are. No one can learn from your success if you merely attribute it to “luck.”

Disclaimer: I was asked to give this talk at the very last minute. I’m happy with the message, but I’m a little disappointed in the delivery. I wish I had more time to prep for my first TEDx presentation.

Next page