High Frequency Data Refresh With Google App Engine

I'm developing a service using GAE for android clients and need to refresh application data on a regular basis, say once a minute. In terms of architecture, this is how the applica

Solution 1:

"it's not suitable for such high frequency tasks (or not even supported)" - this is not exactly correct.

Cron jobs can run at intervals as low as 1 minute, see The schedule format:

The following are examples of schedules:

every12 hours
every5 minutes from10:00to14:00everyday00:00every monday 09:002nd,third mon,wed,thu of march 17:001st monday of sep,oct,nov 17:001of jan,april,july,oct 00:00

If you don't need to run a recurring job at a specific time, but instead only need to run it at regular intervals, use the form:

every N (hours|mins|minutes) ["from" (time) "to" (time)]

So for 1 min interval you can use:

every1 minutes

If you need lower than 1 minute intervals you can use the deferred library - tasks can be delayed from the enqueueing moment with time values specified in seconds:

deferred.defer(do_something_expensive, "Foobie bletch", 12, _countdown=30, _queue="myqueue")

The answer to the last questions really depends on how you want your app to behave: have the data immediately available for the client app when the client app starts or have the client app wait until the backend collects the data.

If you're just relaying the collected data to the client either way is fine, there is no "common practice" (other than driven by higher costs for constant updates, of course). But if you plan to also offer results from processing historic data you'll probably have to go with constant updates (or maybe just during the market open hours).

Update:

The Task Queues are preferable to the deferred library, the deferred functionality is available using the optional countdown or eta arguments to taskqueue.add():

• countdown -- Time in seconds into the future that this task should run or be leased. Defaults to zero. Do not specify this argument if you specified an eta.

• eta -- A datetime.datetime that specifies the absolute earliest time at which the task should run. You cannot specify this argument if the countdown argument is specified. This argument can be time zone-aware or time zone-naive, or set to a time in the past. If the argument is set to None, the default value is now. For pull tasks, no worker can lease the task before the time indicated by the eta argument.