Flickr Image Resolutions

Out of a sample of 6511 random cat images taken from Flickr, 640×640 represents 31% of all image resolutions.

Below is my chart covering the cumulative distribution of the 731 distinct image resolutions:


As we can clearly see, 10% of resolutions make up 75% of images on Flickr.

Also, the average size of a Flickr image is 1.7MB. They recently announced that they have 6 billion images, for a total of 10PB of images. 10PB of images would use 1,700 WD60EFRX 6TB drives, which retail for $300 each (and yes, you can buy 6TB drives off the shelf). The total cost would be $510,000 for a single set of drives, not counting the computers to run them, and different image resolutions, and replica sets, and so forth. Google did a study in 2007 which found that hard drives which are constantly spinning have an expected lifespan of around 5 or 6 years (you have to do some extrapolating to get that, and it’s an estimate). That comes out to 24 expected failures per month, or $7,200 per month in replacements.

We can assume that Flickr has around 10 datacenters, as a sort of fermi approximation (and CDN caching, etc). If the computers to run the drives cost about as much as the drives itself (which seems reasonable), then we get a total bill of $10 million to setup all of the datacenters, plus $72,000 per month in replacement drives, plus $1 million/yr in staff and $1 million/yr in housing.

Amazon S3 sells data for $0.0275/GB/month over 5PB/month. That comes out to $280,500 per month, or $3.4 million per year. Slightly cheaper than the in-house datacenter.

What is the point of all this? Compression. Every 1% of compression that they can eke out of their images saves them at least $27,000 per year, and that’s a number that’s only growing (unless Flickr folds). So if you can manage to make a JPEG 3% smaller, you could make a hundred grand a year for the rest of your life off the savings.

Flickr is but a small fish in a big pond. According to Quora, Facebook currently has 90 billion pictures, 15x the size of Flickr, and adding 6 billion pictures per month. Someone is uploading all of Flickr to Facebook every month. Using our numbers from above, Facebook spends $5 million per month on new hard drives to put pictures on, not counting replacements. A 1% improvement saves them $50,000 per month.

Something to think about, next time you run a large datacenter.


Posted in Uncategorized | Leave a comment

Dear Prior Owner of Eric Foner’s “Give Me Liberty!” Second Edition Textbook

Dear Prior Owner of Eric Foner’s “Give Me Liberty!” Second Edition Textbook:

Your note was received intact, whoever you are. And you’re right.

photo (4)It did make me smile. Thank you.


Posted in Uncategorized | Leave a comment

REST Performance

Rest performance: It’s a measure of how much rest you can get in a given span of time.

REST performance: It’s a measure of how much REST you can get in a given span of time.

Similar, but subtly and importantly different. My tool will measure the second, but not the first.

Do, check it out. Here’s the sample perf.def.js file, broken up into bite-sized chunks with explanations.

var perf_utils = require(PERF_UTILS_PATH);
var mkpoisson = perf_utils.mkpoisson;

Note that the .def.js file is just a NodeJS file, like any other. It gets called with the global PERF_UTILS_PATH variable, which defines the path to the perf utils file. This file contains several one useful utilities, such as mkpoisson. You’ll see how this is useful later.


var ClientInstance = function() {
    this.client_id = "";

This defines what consistant state is. perf-rest works by maintaining a whole bunch of instances, and moving them between different states according to rules that we’ll define below.  One way to think of this is sort of like cookies on a web browser – your web browser may make multiple requests to a web site, but the cookies stay roughly constant. For example here, I’ve defined a variable “client_id” which is a unique ID which the server gives to us.


var ClientRequests = {
    "base": {
	hostname: "localhost",
	port: 3000,
	headers: {
	    "Content-Type": "application/json"

This is where we start defining the different possible requests that can be made to the REST server in question. Note that this is not a complete request: it is only a base request which other requests build upon. Much like an object-oriented language (I realize that my audience is going to be more and more Java-oriented, sadly, so you can think of it like Java where classes can subclass other classes).

    "authenticated": {
	oncall: function(state, params) {
	    this.headers = {client_id: state.client_id};
	parent: "base" // The parent request

Woah! I was just talking about subclassing requests! And here we are. An “authenticated” request has all of the fields of a “base” request.

It also has the oncall function defined. This function is run when the request is run, and is given the instance state. This state is used to create a header. After the oncall function is run, the headers are merged with the headers defined in the base request. Headers are the only thing that are merged; everything else is overwritten. For instance, if the authenticated request had specified a hostname, then it would overwrite the one in it’s parents.

    "login": {
	parent: "base",
	path: "/login",
	method: "GET",
	onfinish: function(res, state) {
	    state.client_id = res.client_id;
	    //console.log("We are client "+state.client_id+"!");

Notice the new onfinish function. This function is called with the parsed response from the request. Note that it’s parsed as JSON: If the remote server doesn’t respond in JSON, then that’s so sad for you. File an issue on Github and I’ll fix it.

The onfinish function, as you can see here, can be used to update state from the server’s response.

    "perform_expensive_action": {
	oncall: function(state, params) {
	    var delayTime = Math.random()*10.0;

	    // Look! You can specify these in the oncall function
	    this.path = "/delay";
	    this.body = {delaytime: delayTime};
	parent: "authenticated",
	// path is specified in the oncall function
	method: "POST"

Hey, you figure it out. Let’s move on to how these requests apply.


…not like Alabama or Texas. Like Markov.

var ClientStates = {
    "preinit": { // Stores the delay for before init is called
	delay: mkpoisson(60.0)

There are three special states, preinit, init, and exit. You will see each of these.

Preinit specifies the delay between when the program starts and when the init state is called. Note here the mkpoisson we noticed before. “delay” can be either a number or a function: If it’s a number, then that exact literal number of seconds is used as the delay. If it’s a function, then that function is called and the return value is used as the number of seconds in the delay. What mkpoisson(x) does is return a function which returns a poisson random number centered on x. That is, by setting delay to be “mkpoisson(60.0)”, we have said that the delay for preinit is a poisson-distributed random number centered on 60.0.

    "init": {
	request: "login",
	transition: [{ // Specify the transition after the request
	    prob: 1.0,
	    dest: "delay_action"

The second of the three special states. When an instance enters the “init” state it is said to be “active” or “alive”, and appears in the output’s instance (“client”) count.

That’s the only special thing about init. Otherwise, it’s just a normal state. “request” specifies which of the above requests gets used. The request gets sent once, when the state is entered. If you don’t want to send a request, use the placeholder “noop”.

“transition” defines which state we transition to after the request is completed. It is a list of probabilities, delays, and destinations. One of these objects is picked out of the list based on the flat probability distribution defined by the “prob”. Note that the sum of the “prob” fields must add to 1, or else undefined behavior occurs.

Once one of the paths is selected, a delay is inserted according to the “delay” field. Obviously, in this instance, the “delay” field is omitted, so the delay is taken to be zero (that is, an immediate transfer to the next state). The “destination” field specifies the name of the state to move to.

    "delay_action": {
	request: "perform_expensive_action",
	transition: [{
	    prob: 0.95,
	    delay: mkpoisson(30.0),
	    dest: "delay_action"
	}, {
	    prob: 0.05,
	    dest: "exit"

Here, we can see the delay field in use. After the request “perform_expensive_action” is performed, there is a 95% chance that after a delay (defined by a poisson distributed random number centered at 30.0) the instance in question will move to the “delay_action” state. Which, by happenchance, is this state. So it’s a loop. With a 5% probability, the instance will immediately move to the “exit” state. While fairly self-explanatory, I feel I should spell it out. When the exit state is reached, the instance is no longer included in the output’s count of instances/clients, and nothing more can be done.

One last thing before I finish this file.

module.exports = {
    Requests: ClientRequests,
    States: ClientStates,
    Instance: ClientInstance,
    numClients: 1000

This exports the requests, states, instance definition, and the number of clients.


Here I’ve described in pretty good detail my perf-rest project’s definition files. The actual usage of the program is described on the Github page. If you have any questions or comments, please feel free to email me or track me down or file an issue with the Github issue reporter. Also, pull requests are cool. Or requests for support. Or just drop a line and say you used it.


Posted in Uncategorized | Leave a comment

3d Photoshop

Recently, a group of researchers at CMU developed a program which uses off-the-shelf 3d models to edit images in 3d. I’m terrible at explaining, so here’s a link to the thing: (also so I don’t lose it…)

Also, new panorama!

The camera mount changed to be a servo ziptied to a block of wood. My suspicion is that the wood absorbs most of the high-frequency interference we saw with the prior panorama.


Posted in Uncategorized | Leave a comment

Captain Crunch

I’m going to start running out of crunch-related names. I better practice my quadcopter flying.

Here’s a telemetry graph of what happens when you lose a propeller mid-flight:


(you may have to click on it to view closer) Note the brownish line, which is the motor 4 output signal. It stays synced (roughly) with the others until 16000 on the X, at which point I lost the prop and the control system pegged the motor throttle to compensate. Then it hit the ground, bounced a couple of times, etc.

On happier news, I’ve been working with RunSFM (, which is a SFM implementation made by Nghia Ho. It’s really quite nice. Here’s a link to a pointcloud of my R/C radio:

Okay, I lied. I can’t get PotreeConverter to work, and Google doesn’t reveal any other pointcloud viewers. It’s saying something about a regex_error. This’ll take some looking into.

That came from about 40 images.

For reference, here’s my timing data with RunSFM on a 4-core 8GB machine:

42 full-res images (3888px wide): 151 minutes by the clock, 343 minutes CPU time.

40 full-res images (3888px wide): 89 minutes by the clock, 214 minutes CPU time.

So a lot of variation, depending on the nature of the images I suppose.


Posted in Uncategorized | Leave a comment


Few things are as stressful as your significant other falling 45 feet and shattering her legs beyond repair. Especially when you caused it.

Let me tell you, it does not do any favors for the relationship.

Thankfully, at least in my case, only the legs broke. The rest of her is apparently happy as a clam despite suffering a more-than-8G impact. The recording instruments measured 8G, but it’s possible the impact was higher:


(the chart covers 8.2 seconds of flight)

I don’t think the bouncing helped any. Ignore the blue and red lines which slope down monotonously. Those demonstrate how the GPS module has issues with rapidly dropping altitudes.

There is scant imagery from my day, since the one successful flight flew a downward-facing camera over a grassy field. Not exactly the height of interesting. At least I have lots of pictures of grass. And a few of my base station:


Checkout my wavy landing pad in the corner. I paid extra for wavy cardboard.

Oh well. Time to checkout the damage:

photo (2)

Okay, it’s a bit hard to see. But note just to the left of the DVD stack (under the chewing gum kit) is a pile of discarded carbon fiber plates which were the landing struts. Right above that you can see the ragged edges of some that are still attached to the arms but broke off. On the right side of the copter note that the legs broke off almost too high to notice.

photo (3)

That was a standoff that separated the two parts of a leg. Yes, those are the threads that were ripped out and are sticking out of either end. Likely not going to use this particular standoff again.

I guess I’m going to get a crash course in building landing gear.

In totally unrelated news, I went to California and had a blast (which is why I didn’t post last week or the week before). We went to an aquarium and I got a shark:

photo (1)He looks less grainy and more adorable in real life.


Posted in Uncategorized | Leave a comment

Speed Enforced by UAV

Got to fly (quadcopter) this weekend. Also got to fly (airplane) this weekend to New Braunfels, which took me directly over the field I fly my quadcopter in.

Speaking of my quadcopter, I’m starting to work on a new vision landing system. So far I just have test imagery of a pad:

Clearly a downward facing camera. The picture below will back me up on this:

But then where did this picture come from?
Note the (illegible) datestamp in the corner, which came from that webcam.

I feel like a downward facing camera wouldn’t have a picture of the sky. I always thought it was just a figure of speech to have eyes on the back of your head – apparently this webcam literally does.

Anyway, the plan is to have the Pi do image processing to detect where the copter is in relation to the pad, and then it’ll tell the Pixhawk to move to align with the pad.

Altogether the Pixhawk, Pi, GPS, 900MHz radio downlink, and webcam pull 0.8A. In case anybody’s wondering. Also, I get 190 seconds of flight time with a 3s 2200mAh. There’s relatively little literature on the web about what to expect, so that’s the number I’ll quote for people who are looking. I can get about 5m/s top speed out of it, so I can fly 950 meters on a full charge. 450 meters if I want it to come back also.

But numbers are hard to grasp, so let’s overlay it on Google Earth. The red circle is my max operational range. The yellow circle is my one-way-ticket range.


North is oriented as up. Please don’t try to stalk me using this picture. I don’t even live here. Nobody lives here. I fly in a fake neighborhood that was constructed to scare away enemy bombers.

Anyway… The neighborhood to the Southwest very nicely fits inside the one-way circle. Coincidence? Probably not. The school district that serves this fake neighborhood has all sorts of rules about busing children more than certain distances to school. (that’s a school that I fly out of in the center, by the way. Two schools sharing a campus, actually)

So, I have a webcam on a quadcopter, what should I do with it? Obviously there’s some really cool stuff I can do. Like play with poles. Or juggle. Or something with LEAP motion. Or build stuff. If you’re taking applications, Zurich, count me in!

Anyway. Right now I just have a panorama to show off:

I was having resonance issues with the camera, so some images have a high frequency wobble in them which is caused by the camera shaking while the webcam’s reading out scan lines. I hadn’t had this issue before Sunday so I don’t know exactly what started causing it, although the winds were a little stronger that day, 20 mph gusting to 25. Enough to make the panorama be at a visible tilt – that tilt is the quadcopter compensating for the wind.

Notice the highway with cars on it. Observe the following two images:



The second image was taken exactly 1 second after the first one. Take special note of the white car/truck thing that’s in front of the black SUV thing that’s moving left-to-right. It’s above one tree in the first image then a different tree in the second, just to the left of center (right where the off ramp is).

According to Google Earth, the distance between the quadcopter and the highway is 277 meters as the crow flies, plus the 100 meters off the ground that I operate my copter at, and the highway goes away from the copter at about 10 degrees. My measurements indicate that the horizontal field of view of my webcam is 49 degrees.

According to GIMP, the truck moved from pixel 270 to pixel 326, or 56 pixels. I accounted for the motion of the UAV manually, but ideally that’d be automated.

The images above are 640 pixels wide, so 56 pixels is 8.75% of the field of view or 4.288 degrees. Note that the copter is 294.5 meters from the section of highway in question, so 4.288 degrees translates to 22.08 meters.

However, remember that the truck is going away from the camera at a 10 degree slant. How do we account for this? We remember Trig. Specifically, the observed motion will equal the cos(angle from perpendicular) multiplied by actual motion. In numbers, 22.08 m/s = cos(10 degrees) * velocity, so velocity = 22.42 m/s or 50.15 mph. Which the speed limit on that segment of road is something like 65 mph. So Anonymous Truck Alice was going (well under) the speed limit!

That was anti-climatic. I was kind of hoping to catch a speeder.

Point is, I could have caught a speeder. If there had been a speeder.

Sometimes there are speeders.

And I will find them.


Posted in Uncategorized | Leave a comment

Teensy/Pixhawk I2C Development

I haven’t gotten a chance to fly (quadcopter) since I last posted. I have, however, gotten a chance to fly (airplane) since I last posted. I think the two are related.

However, I got a Raspberry Pi! It’s running Arch. I named it Helium and put it on my quadcopter, so next week hopefully I have a pi commanding a pixhawk via I2C.

However, only time can tell.


Posted in Uncategorized | Leave a comment

Video Quadcopter

It’s going to be a few weeks before I talk about something other than my quadcopter.

Until then, these two videos:

It’s running a Raspberry Pi to run the camera. My plan is to connect it via I2C to the Pixhawk, and to build a servo gimbal for the camera. From there I dunno.

Also, while I was developing the ground station I ended up with this graph:Screenshot-20

A graph of altitude (centimeters vs. time). I’m unsure how to read this graph.


Posted in Uncategorized | Leave a comment

Every day I’m Quadcopterin’

The song “Party Rock Anthem” by LMFAO has almost 690,000,000 views. What is this world coming to?

But, I got a quadcopter last week! It’s 3D Robotics’ quadcopter kit. Which is a very nice kit, but let me warn you it includes neither a PPM R/C system nor a battery, both of which are required to make it run. So if you get one be sure to get those also, unless you can rewrite the firmware to not have that necessity.

The instructions were clear enough that I was able to put it together in a couple of evenings after work, without screwing anything up. However, getting the software stack to run on Linux took some doing (and all of Saturday). A note for future users: I had good luck with the APM Planner 2.0 software (compiled from source) running the default APM Copter firmware. My receiver is a Spektrum DX6i receiver from back when I flew model airplanes, and the satellite radio works just fine although I will be doing range tests next weekend. At the same time I put a Raspberry Pi and a webcam on it.

As for the battery, for reference, I went with the nano-tech 3 cell 2.2AH LiPo battery, which goes for just under $30 at Hobby Town.

Something to be wary of: The default APM Copter firmware doesn’t have a very good battery monitor (or maybe I misconfigured it), and LiPo batteries dislike being overdischarged. According to the internet once you get below 3V/cell (9V for a 3 cell battery), you just have to toss the battery unless you have a charger that can charge the individual cells at 5 milliamps or whatever.

Here’s what I did, which worked for me although my battery probably doesn’t have as long a life as it could have: I plugged it into my regular charger, set it in the lowest mode (1A), and started a charge cycle. It went for about a minute before it noticed the battery was undervoltaged. I reset the charger, checked the cell voltage and balance using a volt meter, and started a new cycle. This went on until eventually the battery gets back to 9V, at which point I went to the 2 amp mode on the charger and fill the battery up fine.

Of course, I kept a close eye on my battery during this process and you should too.

On a lighter note, I kept on having issues with my quadcopter flipping over:


Hrm… The computer says it’s upside-down…

Turns out that the preflight calibration was messed up, and the onboard gyros were off by 20 degrees.

But, yesterday, I managed to get the GPS-based modes working (specifically loiter and guided). Loiter mode is rock solid, even in 10 knot wind. Kudos to the guys at APM who managed to make that work.


Posted in Uncategorized | Leave a comment