Using the Google App Engine Datastore Remote API with Django

So you have a Django application and you want to inspect the data in your production Google App Engine (GAE) datastore. What are your options? You could use the developers console - it has a query tool - but it's pretty limited, and it doesn't currently provide a way of exporting elements. You could write and deploy a script that collects and prints the data, but who wants to do a deployment every time you want to tweak the script? So what can you do?

Fortunately GAE offers a Remote API for Python for interacting with remote data elements from the comfort of your very own CLI.

For this example, I'll be using a project with the id blhaastest. If you're unsure of your project id, you can find it in your developer console:

First thing to do is verify your Google App Engine symlinks resolves correctly:

> echo $GAE

Next, start the Remote API Shell (Google login required; may require an App Password if 2-factor-authentication is enabled):

$GAE/ -s

If your login is successful, you should get a response that looks like this:

App Engine remote_api shell
Python 2.7.9 (default, Dec 11 2014, 02:36:08)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
The db, ndb, users, urlfetch, and memcache modules are imported.

Now that your remote shell is up and running, you can begin to execute commands against your remote datastore.

First thing to do is import your Django settings, so the shell can begin to know about your application:

> import os
> os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")

Experienced Django developers will recognize this line above as having been plucked from

Next, import the file containing your model definition(s):

> import library.models

In the example above, the model definitions are in a file named, located in the library directory.

You can now begin using QuerySet operations. For example, here's how you can fetch all Books, sorted by title:

> books = library.models.Book.objects.all().order_by('title')

You can print the Book titles like this (Remote API Shell allows multi-line input, terminated by an empty line):

> for book in books:
>     print book.title
> [return]

Results in the following output:

Don Quixote
Invisible Man
Nineteen Eighty Four
The Illustrated Man
Things Fall Apart

Special thanks to GAE documentation for this article, which I used to help get me started. It has more tips and tricks for using the Remote API, so check it out!

> Read More

Be Careful With Default Values for Method Parameters in Python

Today I Learned: python instantiates default values for method parameters only once — at method definition time. This can cause big problems if your method modifies values in these entities, because these modifications will be propagated to subsequent calls to the same method.

I spent the greater part of day yesterday hunting down some weird behavior in a python app, and I thought I'd share. Here is a simplified version* of what I found...

def test_method(self, http=httplib2.Http()):

   # Remember the original request method
   request_orig = http.request

   # Define a closure that executes the original request method
   def new_request():
      resp, content = request_orig()

   # Replace the request method with our own closure.
   http.request = new_request

We're assigning the new_request closure to the http.request method, and never resetting it. A subsequent method call sees this closure as the original method, and everything breaks down.

The fix is to remove the default http value, and always pass in a throwaway http object.

*Note: the real (more complicated version) is in the authorize() method (line 439) here.

> Read More

Installing Python and Django using MacPorts

Use MacPorts to install python (I'm using Python version 2.7 for this demo):
$ sudo port install python27

If successful, MacPorts should give you this informational message: "To make python 2.7 the default (i.e. the version you get when you run 'python'), please run: sudo port select --set python python27" So let's do that:
$ sudo port select --set python python27

If you want to make sure that installation worked, and you have the correct version, check like this:
$ which python
$ python --version
Python 2.7.8

Using your MySQL server command line, create a MySQL database. I'm using the name 'djangodemo' for this example:
> CREATE DATABASE djangodemo

Install pip:
$ sudo port install py27-pip

If successful, MacPorts should give you this informational message: "To make the Python 2.7 version of pip the one that is run when you execute the commands without a version suffix, e.g. 'pip', run: port select --set pip pip27" Let's do that:
$ sudo port select --set pip pip27

Now we can install Django:
$ sudo pip install Django

Verify that Django is installed:
$ python -c "import django; print(django.get_version())"

Add the Python bin directory to your path. Add this line to your .bash_profile:
export PATH="$PATH:/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin"

If you're using MySQL, install the Python/MySQL connector:
sudo port install py-mysql

Now you can start building your project. The official Django tutorial is a great place to start.

> Read More

Installing node.js on a Mac from scratch

Every other month or so I've had the need to provision a clean Mac with a node.js development environment. I've summarized my steps here for quick reference (mostly for myself, but perhaps other people will find this useful).

1. Install Xcode from the Mac App Store.

2. Activate Xcode license: sudo xcodebuild -license

3. Install command line tools: xcode-select --install. This will bring up a prompt that asks you if you want to install the command line tools. Click "Install" to continue.

4. Download and install MacPorts

5. Use MacPorts to install node.js and node package manager: sudo port install nodejs npm

6. Use npm to install npm install (or npm install -g to install it globally)


Locally and Remotely Renaming a Branch in Git

Let's say you're busy working on your project, creating feature branches right and left to build out your super sweet app. Then your dev team decides to change their feature branch naming convention. How do you proceed without having an aneurysm? Can you easily accomodate your team's plan and not end up wanting to punch people in the face? Yep! As a matter of fact it's super simple to rename your branches. Here's how.

> Read More

Update to Installing Yii Users and Rights

It has been a year since I originally wrote the "Installing Yii Users and Rights in 5 Steps" tutorial, and both the Yii framework and the modules themselves have had some updates since then. I made some slight tweaks to the tutorial and updated the git repository to match the current builds.

You can view the updated post containing the tutorial here:


Controller Access Control With the Yii Rights Module

It's easy enough to determine how the Yii Rights module works when used to perform inline access control checks in your code. You simply call ($user->checkAccess($operation) === true), and the internal details are easy enough to dig into. But how does it work with your controllers to allow or deny access to a page (or more specifically, an action in your controller)?

If you haven't yet read up on access control filters, you should check out the official Yii Controller Basics first and get familiar with filter chains. Then come back and read the rest of this post.

> Read More

Displaying Yii's CLinkPager "First" and "Last" Buttons

Yii provides an excellent set of tools for displaying paged data, namely the CListView and
CGridView widgets. These widgets are easy to use, are highly customizable, and provide the kind of pager that is in use all over the web. Which makes it all the more strange that the pager doesn't work quite as expected.

The strange default behavior is that built-in css attributes hide the "First" and "Last" buttons. In fact, if you're like me, you didn't even know that the pager had a "First" and "Last" button until you started looking into putting ones in there. If you view the page source and scroll down to the pager, you'll see these elements, each of them contained within <li class="first"> and <li class="last"> tags, respectively. These tags are targeted by a display:none css attribute contained in the default pager stylesheet.

I have no idea why these are hidden by default, but it's not too much work to make them visible.

Read on to see how to change:


> Read More

Using Yii ActiveRecord to Assign from a List of Values

Often times you'll have a database table containing a list of values, and you want to assign one of those values to another entity. Imagine you're assigning an author to a book, or a currency type to a sale amount, or a country preference to a user. In all these cases you have one table/class representing a list of values, and one table/class that stores, among other things, a reference to a value/instance in the first table/class.

Luckily, Yii makes it easy for you do implement this.

For this example we're going to focus on books and authors, because everyone still reads books, right?

> Read More

Updating to the Latest Yii Framework

The good folks at Yii dropped their latest framework release, 1.1.9, on January 1 of this year. Updating my applications use to this new framework was comically simple thanks to a small trick: using a symbolic link to reference the framework.

Whether you're downloading Yii for the first time, or upgrading to the latest Yii framework, you can create a symbolic link in your web root directory like this:

ln -s yii-1.1.9.r3527/ yii

Also update the following webapp files and replace each reference to the hard link with a reference to the symbolic link:

  • /webapp/index.php
  • /webapp/index-test.php
  • /webapp/protected/yiic.php
  • /webapp/protected/tests/bootstrap.php

Now here's where this comes in handy: when you're ready to upgrade to a new Yii framework, simply download it and unzip it in your web root, and update the symbolic to point to the new framework. Boom, thirty seconds, you've upgraded.

(Of course it's only this easy if you haven't hacked any core framwework files. You haven't hacked any core framework files, right?)

Syndicate content