Category Archives: hacking

Three-day monk

Sometimes I start something passionately, dedicate myself to the cause, and fully immerse myself in an activity. It can be anything – starting a new project, doing a proof-of-concept, or starting an essay for example. If a goal can be reached within one or two days, I often succeed. However, if the goal requires more time, it often drags for weeks, or years. More often it makes for an abandoned project.

Japanese have a phrase: 三日坊主(みっ・か・ぼう・ず)which roughly translates as three-day monk. It means exactly that. It means I am hyped up, fully committed, fully dedicated, but for a very short time. Only to fall back to my old ways and habits, and never achieve what I’ve set myself to achieve.

The way to avoid the three-day monk trap, is to dedicate  some time to the activity every day. To make it a daily habit. It does not need to be long amount of time – it can be short. But consistently bring the activity to the forefront of your mind, and to do it. Keep doing it. If you plan to blog or start new project – go to your blog and write a draft, or a beginning of a draft, or a first sentence. Start. Next day come back, add to it. Next day add more. You do not have to spent hours or days on an activity. You do not have to immerse yourself for extended periods of time. Doing it more frequently and consistently will bring better results.

 

Photo metadata analysis

Overview

Digital SLRs record a number of information (metadata) about the shot being made together with the actual image. This includes for example the aperture, shutter speed and the focal length that were used during the shot. This information is stored in an Exchangeable Image Format (EXIF) datastructure, that we can read, and process. If you have a large collection of photos, you can learn about certain facts that you may not know otherwise. I’ve done, for example, an analysis of my collection in the context of focal length I use, before deciding on what prime lens I will purchase.

Attempt 1 – nodejs with exif module

If you are not into nodejs and coffeescript, you may skip this section entirely.

Prerequisites:

  • nodejs
  • coffeescript
  • exif module (npm install exif)
  • fs module (npm install fs)
The script below can be executed and it takes variable number of arguments in a form Directory1 Directory2 …
The script will run through your directory and generate a comma-saparated-value file, with a date, aperture, focal length and exposure time, that then could be imported into Excel or used in Gnuplot.

ExifImage = require('exif').ExifImage
fs = require 'fs'

process_image = (img_filename) ->
 console.log 'Processing file: ' + img_filename
 new ExifImage { image : img_filename }, (err, img) ->
 if err
   console.log 'Error: ' + err.message
 else
   process.stdout.write tag.value for tag in img.exif when tag.tagName is 'DateTimeOriginal'
   process.stdout.write ', ' + tag.value for tag in img.exif when tag.tagName is 'ExposureTime'
   process.stdout.write ', ' + tag.value for tag in img.exif when tag.tagName is 'FocalLength'
   process.stdout.write ', ' + tag.value for tag in img.exif when tag.tagName is 'FNumber'
   process.stdout.write '\n'

process_directory = (dir_name) ->
 fs.readdirSync(dir_name).forEach (file) ->
 extension = file.split('.').pop()
 if extension.toLowerCase() == 'jpg'
 process_image dir_name + file

a = process.argv[2..]
a.forEach (val, index, array) ->
 console.log 'Processing directory: ' + val
 process_directory val

Unfortunately, even for a directory that contains 20 files this will not work. Your script will chew all available memory and crash. It will work for a directory with only few files though. Try it.

TODO: check how the existing exif module is implement it and see if it is possible to make it only use the EXIF record instead of loading the entire image into the memory.

Attempt 2 – bash and awk

After failing with the cool coffeescript-based histogram-drawing attempt, I’ve decide to quickly hack a command line bash script that would together with awk and command line exif program, achieve what I want: a histogram of all focal lengths used in a photo collection. This method is fast and reliable.

Prerequisites:

To do that for your collection, here’s what you’ll need

  • bash
  • awk
  • exif (or any other command line executable tool you fancy, that can read EXIF data)
  • (optional) gnuplot (command line plotting tool); you can use Excel or similar program, too.

If you do not have exif installed, and you are on macosx, to get exif you say: brew install exif (if you do not use homebrew, you should definitely give it a try; you’ll love it.).

If you on Linux, use your package manager and install exif. bash and awk come as default on both macosx and Linux.

First, we need to collect all the focal length from all the images in a given location. The following, will go into your specified DIRECTORY, and search for all files in the top level and all subdirectories. The command assumes that you store only images – if you want to limit it to .JPG or .jpg files only, check the manual for find command:

find DIRECTORY -type f -exec exif -m -t 'Focal Length' {} \; > mydata.txt

After that, you can have a look, and mydata.txt should contain something along the lines:

25.0 mm
20.0 mm
26.0 mm
16.0 mm
19.0 mm
39.0 mm
34.0 mm

Great. Now, we need to convert it to a histogram-like structure, so that we can visualize it on a graph. To do that, we’ll use cat and awk:

cat mydata.txt | awk '{count[$1]++} END {for (j in count) print j, count[j]}' > mydata

The above line will count each occurance of a given focal length, and combine it into a table. If you look inside mydata file, you should see something like that:

29.0 22
20.0 44
150.0 1
187.0 1
105.0 13
31.0 15

Now, with the mydata file, we’re ready to use gnuplot (or you can load the file into Excel). Simply fire up gnuplot, and do:

plot "mydata" w impulses lw 3 lc rgb "#00AA00"

Done. I hope you’ll enjoy checking what is the focal length you love to shoot at.

Check examples of focal length histograms of my photos.

 

Programming: the art of writing

To some people, especially those who cannot program computers, the art of computer programming seems more like “monkey and typewriter” type of activity. This misconception is far from the truth. Computer programming is a complex and highly rewarding intellectual activity, much closer to writing a novel, than conducting an engineering task. The art of computer programming always appealed to me — it is a vast land somewhere on the intersection of science, mathematics and artistic expression of myself. Focus, creativity, discipline, and an inner sense of beauty, all have to come together, for the final piece to be created.
Sure, there is a lot of bad software around. Same as there is a lot of bad novels or films. Especially those created by committees and executive boards. There is, however, a growing number of small indie developers writing high quality, beautiful software.

Stephen King said: “The scariest moment is always just before you start.

Programming is a skill, a tool, that you can use to express yourself. It is a way of being in the world. How to get started? First, you have to master the craft itself. Simply write. Write code. Write software. For yourself, for your mom, for your friends. Keep writing. Anything. In any language there is. There is no other way but doing it. Start small. Tasks that can be achieved in under an hour will work for you like etudes for musicians. Practice.

“Talent renders the whole idea of rehearsal meaningless; when you find something at which you are talented, you do it (whatever it is) until your fingers bleed or your eyes are ready to fall out of your head. Even when no one is listening (or reading, or watching), every outing is a bravura performance, because you as the creator are happy. Perhaps even ecstatic.”

Then, once you grow and become more confident, plan bigger. Plan a novel or plan a symphony. There is lots of parallels between writing computer code and the act of writing, in general. You need to have the concept in your head. You need to feel it, you need to know what it is. Then, you need to organise a routine, a discipline, and let the creativity take over.

An excellent and accomplished writer, Stephen King, has written a spectacular book “On Writing: a memoir of the craft“. The book provides many insights into the process of being creative, and how to write. It applies to computer programming, too.

“There is a muse, but he’s not going to come fluttering down into your writing room and scatter creative fairy-dust all over your typewriter or computer. He lives in the ground. He’s a basement kind of guy. You have to descend to his level, and once you get down there you have to furnish an apartment for him to live in. You have to do all the grunt labor, in other words, while the muse sits and smokes cigars and admires his bowling trophies and pretends to ignore you. Do you think it’s fair? I think it’s fair. He may not be much to look at, that muse-guy, and he may not be much of a conversationalist, but he’s got inspiration. It’s right that you should do all the work and burn all the mid-night oil, because the guy with the cigar and the little wings has got a bag of magic. There’s stuff in there that can change your life. Believe me, I know.”

The advice on making a first prototype (or first book draft as he calls it):

“I believe the first draft of a book — even a long one — should take no more than three months…Any longer and — for me, at least — the story begins to take on an odd foreign feel.”

And, how te evaluate your proof-of-concept:

“Write with the door closed, rewrite with the door open. Your stuff starts out being just for you, in other words, but then it goes out. Once you know what the story is and get it right — as right as you can, anyway — it belongs to anyone who wants to read it. Or criticize it.”

Deploying nodejs application on Apache web server

In this post I will explain how to deploy your own NodeJS application on Apache server. I’ll also briefly discuss how to use supervisor to manage your multiple node-driven applications.

There are many possible ways for deploying your node apps and redirecting input/output streams from a node process to your Apache/nginx web server. In this post we use a simple python tool called: supervisor. To use supervisor, we need to setup a startup script for our application, install and configure supervisor, and configure VirtualHost on the apache or nginx web server.

start.sh script
Create a simple Bash script, called start.sh, in the root directory of your application. This script will start up your application.

#! /bin/sh
cd /var/www/vhosts/{my_first_app.com}/{path_to_my_first_app}
/usr/local/bin/node app.js

Make sure that the path to node and the path to your application folder are correct. Test by executing the start.sh script.

supervisor install and configuration
supervisor is a tool that provides an easy way for managing processes. Installing supervisor is easy. You simply follow your standard python package manager installation procedure: eg. easy_install supervisor or pip install supervisor

To configure supervisor first generate the standard configuration file:
echo_supervisord_conf > /etc/supervisord.conf

and then fill-in appropriate fields to match to your application paths:

[program:my_first_app]
command=/var/www/vhosts/{my_first_app.com}/{path_to_my_first_app}/start.sh
autostart=true
autorestart=true
startretries=3
stdout_logfile=/var/www/vhosts/{my_first_app.com}/{path_to_my_first_app}/log/server.log
stdout_logfile_maxbytes=10MB
stdout_logfile_backups=5
stderr_logfile=/var/www/vhosts/{my_first_app.com}/{path_to_my_first_app}/log/error.log
stderr_logfile_maxbytes=10MB
stderr_logfile_backups=5

To launch the supervisor you run supervisor command. To check the status of your managed application, use supervisorctl These executables live in your python path (where your particular python package manager installed them).

Apache VirtualHost configuration
Inside your VirtualHost entry, setup the following:

ServerName myfirstapp.com:80
ServerAlias *.myfirstapp.com
DocumentRoot /var/www/vhosts/my_first_app.com/my_first_app/public
AllowOverride FileInfo Indexes
RewriteEngine On

# redirect all non-static requests to node
RewriteCond %{DOCUMENT_ROOT}/${REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ http://#{server_ip}:#{application_port}%{REQUEST_URI} [P,QSA,L]

 

Full deployment automation
With the above instructions you have to manually copy, build and deploy your application into the appropriate directory on the server, in the examples above, at /var/www/vhosts/{my_first_app.com}/{path_to_my_first_app}

This is quite a lot of work, and ideally, a single push to a remote repo should update your application deployment on your production web server. To achieve that, we need automation. I will explain it in the next post – stay tuned.

Advanced topics for Android developers

Google I/O talk notes.

Dealing with various APIs and Android versions.
Coding patterns.

How to deal with various device capabilities.

private static boolean isNewAPI =
   android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.HONEYCOMB;

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    Intent intent;
    if (isNewAPI) {
        intent = new Intent(this, ModernActivity.class);
    } else {
        intent = new Intent(this, LegacyActivity.class);
    }
    startActivity(intent);
    finish();
}

Sensor registration example (Orientation Sensor).

private static boolean isNewSensorAPI =
   android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.CUPCAKE;

boolean gyroExists = getPackageManager().hasSystemFeature(
  PackageManager.FEATURE_SENSOR_GYROSCOPE);

IOrientationSensorListener myListener;
if (gyroExists) {
    myListener = new GyroOrientetationSensorListener();
} else if (isNewSensorAPI) {
    myListener = new AccOrientationSensorListener();
} else {
    myListener = newAccOldOrientationSensorListener();
}

myListener.setOrientationChangeListener(myOCListener);

Getting users feedback

try {

} catch (Exception ex) {
   String exText = "something went wrong";
   MyApplication.getInstance().tracker().trackPageView(exText);
}

A – B user testing on Beta deployment. Getting real users feedback on different possible scenarios/visuals/etc.

private static final boolean isA =
UUID.randomUUID().getLeastSiginifcantBits() % 2 == 0;

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    if(isA) {
       setContentView(R.layout.mainA);
       MyApp.getInstance().tracker().trackPageView("/AUser");
    } else {
       setContentView(R.layout.mainB);
       MyApp.getInstance().tracker().trackPageView("/BUser");
    }
}

Using Android Market for beta testing:
- private betas that require login or passcode
- public betas with disguised or obscured listing
- update listing or create new on launch
- point beta users to official app at launch
- be clear that this is beta. venues for feedback
- make sure you translated your app name and description to local languages

Other hints:
- upload your package before distributing it to anyone
- backup your keystore
- use generic gmail account for app releases (sharing the responsibilities)

Sensor orientation tip

int x = AXIS_X;
int y = AXIS_Y;

case (Display.getRotation()):
    Surface.ROTATION_0: break;
    Surface.ROTATION_90: x = AXIS_Y; y = AXIS_MINUS_X; break;
    Surface.ROTATION_180: y = AXIS_MINUS_Y; break;
    Surface.ROTATION_270: x = AXIS_MINUS_Y; y = AXIS_X; break;
    default: break;
}
SensorManager.remapCoordinateSystem(inR, x_axis, y_axis, outR);

Generating unique identifier (per user/device). Use the following method of generating the unique id and store it in users’s preferences.

UUID. randomUUID().toString().

How to get the up-to-date location without draining the battery?
Use passive location provider, or, alternatively, use Location change intent.

final int resultCode = 0;
final String locAction = "com.ioApp.LOCATION_UPDATE_RECEIVED";
int flags = PendingIntent.FLAG_UPDATE_CURRENT;
Intent intent = new Intent(locAction);
PendingIntent pi = PendingIntent.getBroadcast(this, resultCode, intent, flags);
locationManager.requestLocationUpdates(provider, minTime, minDistance, pi);

....

BroadcastReceiver locReceiver = new BroadcastReceiver() {
  @Override
  public void onReceive(Context ctx, Intent intent) {
    String key = LocationManager.KEY_LOCATION_CHANGED;
    Location location = (Location)intent.getExtras().get(key);
    // process new location
  }
};
IntentFilter locIntentFilter = new IntentFilter(locAction);
registerReceiver(locReceiver, locIntentFilter);

To use intents to passively detect location changes start a service to refresh the data without updating a UI.

 receiver android:name=".locReceiver" android:enabled="true"
 intent-filter>
  action android:name="com.ioApp.LOCATION_UPDATE_RECEIVED"/>
  /intent-filter>
 /receiver

Wake alarms and non-waking alarms example:

int wake = AlarmManager.ELAPSED_REALTIME_WAKEUP;
int sleep = AlarmManager.ELAPSED_REALTIME;
long minInt = AlarmManager.INTERVAL_HALF_DAY;
long bestInt = AlarmManager.INTERVAL_HALF_HOUR;
long trigger = SystemClock.elapsedRealtime() + bestInt;

alarms.setInexactRepeating(wake, trigger, minInt, alarmIntent);
alarms.setInexactRepeating(sleep, trigger, bestInt, alarmIntent);

Make your data updates smart by monitoring number of elements. For example:

monitor connectivity

Connectivity Manager cm = (ConnectivityManager)ctx.getSystemService(Context.CONNECTIVITY_SERVICE);
NetworkInfo activeNet = cm.getActiveNetworkiInfo();
boolean isConnected = activeNet.isConnectedOrConnecting();
boolean isMobile = activeNet.getType() ==
   ConnectivityManager.TYPE_MOBILE;

monitor power and battery

IntentFilter bf = new IntentFilter(Intent.ACTION_BATTERY_CHANGED);
Intent bat = ctx.registerReceiver(null, bF);
int bstat = bat.getIntExtra(BatteryManager.EXTRA_STATUS, -1);
boolean power = bstat == BatteryManager.BATTERY_STATUS_CHARGING ||
    bstat == BatteryManager.BATTERY_STATUS_FULL;

monitor docking status and other possible state-change broadcasts to monitor:

ACTION_DOCK_EVENT
ACTION_BATTERY_LOW
ACTION_POWER_CONNECTED
ACTION_POWER_DISCONNECTED
CONNECTIVITY_CHANGED

Cloud-based Android Backup service.

Backing up shared preferences

public class MyPrefsBackupAgent extends BackupAgentHelper {
  static final String PREFS = "user_prefs";
  static final String PREFS_BACKUP_KEY = "prefs";

  @Override
  public void onCreate() {
    SharedPreferencesBackupHelper helper =
       new SharedPreferencesBackupHelper(this, PREFS);
   addHelper(PREFS_BACKUP_KEY, helper);
  }
} 

in Manifest
application android:label="MyApp" android:backupAgent="MyPrefsBackupAgent">
  meta-data android:name="com.google.android.backup.api_key"
          android:value="MyAPIKey" />
/application>

Making your apps more adaptive. Use appropriate keyboard for EditText entry UI.

eg, in EditText  ....
   android:inputType="phone"
   android:imeOptions="actionSend | flagNoEnterAction"
/>

Listen for those special action keys in your OnEditorActionListener:

EditText.OnEditorActionListener myActionListener =
 new () {
   @Override
   public boolean onEditorAction(EditText v,
     int actionId,
     KeyEvent event) {
       if (actionId == EditorInfor.IME_ACTION_SEND) {
         //  handle the SEND action
         return true;
       }
       return false;
    }
};

editText.setOnEditorActionListener(myActionListener);

Handle Audio nicely (interoperate nicely with audio focus).

Make everything asynchronous and make use of background threads (Handler, AsyncTask, IntentService, AsyncQueryHandler, Loader and CursorLoader).

CursorLoader example

// within onCreate...
getLoaderManager(),initLoader(0, null, null);

// Callbacks
public Loader onCreateLoader(int id, Bundle args) {
    Uri baseUri = MyContentProvider.CONTENT_URI;
    return new CursorLoader(getActivity(), baseUri, null, null, null, null);
}

public void onLoadFinished(Loader loader, Cursor data) {
   mAdapter.swapCursor(data);
}

public void onLoaderReset(Loader loader) {
    mAdapter.swapCursor(null);
}

Testing/Debugging. Use Strict Mode:

public void onCreate() {
 if (DEVELOPER_MODE) {
   StrictMode.setThreadPolicy(
     new StrictMode.ThreadPolicy.Builder()
              .detectDiskReads()
              .detectDiskWrites()
              .detectNetwork()
              .penaltyFlashScreen()
              .build());
 }
 super.onCreate();
}

Cleaning up MacPorts

First, let’s see how much macports use up on my system:

$ du -sh /opt
8.1G

Next, how many packages do I have installed in total:

sudo port installed | wc
     538

And how many packages are active:

sudo port installed | grep active | wc
     253

Now let’s trim the entire collection, by uninstalling all inactive packages:

sudo port uninstall inactive

If you have some dependency issues, add ‘-f’ flag to the port command that will ignore dependencies.

This reduced the disk usage in /opt to 4.8G, and got the number of all installed ports to 254.

More cleanup:

sudo port clean --all installed

This removes all temporary files generated while building and installing packages. After that, I got down to 3.6G in /opt. Not bad down from 8.1G.

The world of ListView

The notes taken out of the Google I/O talk “The world of ListView“.

Due to performance and memory considerations, the ListView views hierarchy is fixed, nothing that is not displayed on screen is not part of the hierarchy. The item views are recycled.

Terminology

  • index (refers to the child view),
  • position (refers to the Data in Adapter),
  • id (refers to the unique identifier for data).

Stable IDs:

  • if hasStableIds() == true, the ListView can do a number of tricks to presentation and handling of the items.
  • An ID always refers to the same value.
  • Helps ListView.

Adapters

  • getView(int position, View convertView, ViewGroup parent)
  • full data presentation control
  • Potential optimisation through the re-use of convertView

getView()

the slow way

public View getView(int position, View convertView, ViewGroup parent) {
  View item = mInflater.inflate(R.layout.blabla, null);
  ((TextView) item.findViewById(R.id.text)).setText(DATA[position]);
  ((ImageView) item.findViewById(R.id.icon)).setImageBitmap(
    (position &1) == 1 ? mIcon1 : mIcon2);

  return item;
}

the correct way

public View getView(int position, View convertView, ViewGroup parent) {
  if (convertView == null) {
    convertView = mInflater.inflate(R.layout.blabla, parent, false);
  }
  ((TextView) convertView.findViewById(R.id.text)).setText(DATA[position]);
  ((ImageView) convertView.findViewById(R.id.icon)).setImageBitmap(
    (position &1) == 1 ? mIcon1 : mIcon2);

  return convertView;
}

the Fast Way

create a special datastructure in your app, eg.

static class ViewHolder {
  TextView text;
  ImageIcon icon;
}

public View getView(int position, View convertView, ViewGroup parent) {
ViewHolder holder;
if (convertView == null) {
  convertView = mInflater.inflate(...
  holder = new ViewHolder();
  holder.text = (TextView) convertView.findViewById(R.id.text);
  holder.icon = (ImageView) convertView.findViewById(R.id.icon);
  convertView.setTag(holder);
} else {
  holder = (viewHolder) convertView.getTag();
}

  holder.text.setText(DATA[position]);
  holder.icon.setImageBitmap(...

  return convertView;
}

Comments

  • do not change convertView structure (use view type system for this instead)
  • handling data changes: use the adapter properly, then notifyDataSetChanged() (for new or updated data); notifyDataSetInvalidated() (in case there is no more data available).

Built-in view type system

  • getItemViewType, type of view for a given position. Used to provide the right convertView
  • getViewTypeCount (how many different types to expect)

Slow data source

  • Adapter modifications on the UI thread
  • Fetching data can happen enywhere, request data on another thread
  • Commit adapter changes on the UI thread
  • Call notifyDataSetChanged() in the same UI thread.

Item properties

  • Enabled: item can be selected, clicked
  • Disabled: section dividers/headers within content
  • Single choice mode (radio buttons)
  • Multiple choice mode (checked items)
  • Focusable items (or rows)

Headers and Footers

  • ListView.addHeaderView()
  • ListView.addFooterView()
  • must be called before setAdapter()
  • isSelectable == Adapter.isEnabled() (see below)
  • if you use header/footer, and then list.setAdapter(myAdapter); list.getAdapter() != myAdapter;  Internally, ListView wraps one adapter in another one.

Selectors

  • used to highlight the selected item
  • Not shown in touch mode (there is no selection in touch mode!)
  • Shown behind list items by default. Use android:drawSelectorOnTop=”true” if you want to change it.

Other features, DOs and DON’Ts

  • android:transcriptMode, used to control the behaviour of the scrolling of the list when content is added.
  • android:stackFromBottom, stacks items in reverse order, starts with the last item from the adapter
  • these are useful for chats, messaging, talk, irc, etc.
  • android:textFilterEnabled=”true”, filters and presents only the items that match what’s typed, you need to re-implement the Filter instance.
protected FilterResults performFiltering(CharSequence prefix)
protected void publishResurts(CharSequence constraint, FilterResults results)
  • for optimisation and efficient rendering purposes, all ListView items are turned into bitmaps, with a default black background. Use android:cacheColorHint=”myBackgroundColor”, or set it to #0 to disable it.
  • Use android:smoothScrollbar=”false” to avoid scrollbar changing size when scrolling quickly through a list.
  • DON’T – never set android:layout_height=”wrap_content”
  • DON’T – never use ListView inside a ScrollView
  • It is ok to have horizontal scroller with ListView using vertical scroller though.

C2DM: building push applications

Notes from Google I/O talk “Building push applications for Android“. How to keep the data on the device fresh? There are two techniques: polling and pushing.

Polling is simple to implement. Device must periodically ask server for new data. This is appropriate for situations where data changes constantly (stock quotes, news headlines). The drawback is that radio draws a lot of power, and must stay on for several seconds. Power consumption on a device not actually polling data, is between 5-8mA. However, any active network access increases the power consumption to 180-200mA. Note also Tx is more expensive than Rx. On average, each poll costs roughly 0.5mAh. This gives 144mAh/day for 5m update frequency, and 48mAh/day for 15m updates. (Battery capacity could be between 900-1400mAh).

Pushing uses network only when necessary. Google Contacts, Calendar and Gmail sync and android market, all use of push. It is a bit more complex to implement. Solution: Android Cloud to Device Messaging.

Android Cloud to Device Messaging

  • Requires Froyo 2.2 and Market
  • will be available to all developers
  • Uses existing connection for Google services
  • Allows servers to send lightweight data messages to apps
  • Tell app new data available
  • Intent broadcast wakes up app
  • App supplies UI, e.g. notification if necessary
  • Best effort delivery
  • Heartbeat implementation to re-establish broken connections
  • SSL data delivery implemented

C2DM lifecycle:

App registers with Google, gets registration ID

// Use the Intent API to get a registration ID
Intent regIntent = new Intent("com.google.android.c2dm.intent.REGISTER");
// Identify your app
regIntent.putExtra("app", PendingIntent.getBroadcast(this, 0, new Intent(), 0);
// Identify role account server will use to send
regIntent.putExtra("sender", email_of_sender);
// start the registration
startService(regIntent);

App sends registration ID to its App Server.

  • App will receive the ID as an Intent com.google.android.c2dm,intent.REGISTRATION.
  • Service may issue new registration ID at any time, and the app will again receive REGISTRATION intent broadcast. app must update server with new ID then.
// Registration ID received via an Intent
public void onReceive(Context ctx, Intent intent) {
  String action = intent.getAction();
  if ("REGISTRATION".equals(action)) {
    handleRegistration(ctx, intent);
  }
}

private void handleRegistration(Context ctx, Intent intent) {
  String id = intent.getExtra("registration_id");
  if ((intent.getExtra("error") != null) {
    // registration failed. Try again later.
  } else if (id != null) {
    // Send the reg ID to the app server (in separate thread)
  }
}

App server sends (authenticated) message to Google

Google sends message to device

  • Device receives message, converts it to Intent
    App woken up/started by Intent broadcast
  • com.google.android.c2dm.intent.RECEIVE
  • data.<key> set as intent extras
  • app needs com.example.app.permission.C2D_MESSAGE
public void onReceive(Context ctx, Intent intent) {
  String action = intent.getAction();
  if ("_RECEIVE".equals(action)) {
    // grap a wakelock, use IntentService to do work
  }
}

App can unregister ID (when user no longer wants push)

The device can be offline for a period of time, and a number of messages might be issues during that time. To avoid message explosion, there is a simple mechanism that handles this situation:

Collapse keys

  • latest message replaces older ones with same key
  • Avoids message explosion for offline device
  • App may use multiple collapse keys (max of four in flight per device)
  • State should be in app server, not in message (tell app when it should fetch data)

Attenuation

  • Messages may not be delivered to device immediately
  • Protects devices that are receiving many messages (avoid constant radio wake up)
  • Attenuation per app/collapse key
  • This is done automatically by the Connection Server

Delay While Idle

  • Device tells Connection Server when screen is on and off. Screen off means device is idle.
  • Apps can request message only be delivered when active. This avoids waking up device with info that will not be seen/used (think chat presence, friend location updates, etc).

Demo applications (open source examples)

  • Chrome to phone: a simple extension to Chrome that allows any webpage to be pushed onto a mobile device.
  • JumpNote: notes, with two way push sync, App Engine backend, GWT UI, uses Sync Framework, uses C2DM

Question session

Only Froyo 2.2 onwards, no back porting, heartbeats might be in an order of 30min or so, connection is amortized accross multiple applications (Contacts, GMail, Sync, etc), OMA Push service is a different paradigm, and is not integrated wit C2DM. Security: persistent connection is encrypted with SSL. Generally, do not pass STATE in your massages, push the semantics and state up to the application layer. Max size of the message is 1kB, and the message is retained on the servers for weeks (perhaps months).

REST client application patterns

The notes taken from the Google I/O talk on “Android REST client applications“. There is a number of reasons why to implement a native REST application on a mobile device instead of using a mobile website and app in a browser. Some of the reasons:

  • better integration with the content providers of the mobile platform
  • access to sensors and other applications (Services, Intents, etc)
  • ability to augment the functionality of the RESTful application with other applications on a device
  • much better UI integration
  • simply: users prefer native apps to running apps in a browser

Implementation patterns

Service API pattern

Using this pattern, we wrap all the REST interactions in a worker thread on a service.Provide a processor that interacts with a local Content Provider, and provides a local cache for the REST resources. Add two extra columns: status column and result column. Every resource has a transition state, and we need to state (STATE_POSTING, STATE_UPDATING, STATE_DELETING, STATE_OK). The result column contains the HTTP return code for the last REST operation.

Implement a ServiceHelper, singleton, that exposes a simple asynchronous public API to be used by the user interface. Prepare and send the Service request on the forward path:

  • check if the method is already pending (check the state)
  • create the request Intent
  • add the operation type and a unique request id
  • add the method specific parameters
  • add the binder callback
  • call startService(Intent)
  • return the request id

On a return path: Dispatch callbacks to the user interface listeners

Handling REST Method in an Activity:

  • onResume() add the listener
  • onPause() remove the callback
  • consider these cases: callback comes back when (A) activity is active (B) activity is paused, and return AFTER the callback is made (C) Activity is paused/resumed while the method was running.
  • CursorAdapter handles the ContentProvider notification by implementing a ContentObserver.

For performance considerations, use these:

  • Froyo uses org.json API, with much improved performance. Use it.
  • Enable the gzip content encoding when possible.
  • Use the Apache HTTP client (not the Java one)
  • Be nice, shutdown the Service if there is no requests queued
  • Implement a queue for downloads (if you need to download multiple things in parallel)

ContentProvider API pattern

Similar to the previous model, however, the Activity interacts with the ContentProvider, that works as a facade and hides all the REST Service interactions behind the scenes. It is now the ContentProvider that interacts with Service Helper.

In this pattern the retries have to be internally handled by the ContentProvider.

(Simple) ContentProvider + SyncAdapter pattern

Variant of the ContentProvider pattern. SyncAdapter helps to synchronise remote and local content. The SyncManager implements a queue of SyncAdapters, and manages system-wide syncing. SyncAdapter with the help of SyncManager. It will re-try with incremental delays upon sync failures.

This pattern may not update things as fast as the manual approaches above. (And, even though GMail and Calendar are using it, the Google Twitter Client is probably not using it, right?)

Conclusions

  • Do not implement REST methods inside Activities
  • Start long running operations from a Service
  • Persist early and persist often
  • Cursors can only hold of up to 1MB of data. Minimise amount of data you store in Cursor.
  • Minimize network usage
  • Always use paging (hopefully the REST API supports paging)
  • New in Froyo: Android Cloud to Device Messaging, push notifications (will safe lots of battery life)

Android: Performance considerations & profiling

Notes from the Google I/O talk “Writing zippy Android Apps“.  The example code for the talk is at the code.google.com/zippy-android/

Terminology

Jank – chrome team’s term for stalling the event loop thread. Jank – not being immediately responsive to input. Eliminating jank: reacting quickly to events.

ANR – Android not responding. Your application is badly written. This is the ultimate result of jank. ANR will happen:

  • main thread (event/UI thread) doesn’t respond to input event in 5 seconds
  • a BroadcastReceiver doesn’t finish in 10 seconds
  • typically, eg. when doing network operations in main thread, doing slow disk operation, SQL query, in main thread
  • less than 5/10seconds, users experience a janky/sluggish/slow app behaviour

Performance numbers for Nexus One:

  • 0.04ms – writing a byte on pipe process (or reading simple /proc files from dalvik)
  • 0.12ms – void/void Binder RPC call
  • 5-25ms – uncached flash reading a byte
  • 5-200+ms – uncached flash wrting tiny amount. The more full the flash disk is, the slower the writes become.
  • 16ms – one frame of 60pgs video
  • 41ms – one frame of 24fps video
  • 100-200ms – human perception of slow action
  • 100-800ms – ping over 3G. Varies!
  • 1-6seconds – TCP setup + HTTP fetch of 6k over 3G

SQLite performance:

  • lots of writing. avoid any writing in the UI thread
  • use indexes (use: EXPLAIN and EXPLAIN QUERY PLAN in sqlite to see what’s your query doing. use the sqlite_wrapper.pl script to profile your queries)
  • cat /proc/yaffs (check the nBlockErasures – very slow, check nPageWrites nPageReads)
  • log files: often much cheaper to append to a file than use a database

android.os.AsyncTask

  • overwrite with your own task class.
  • constructor takes 3 parameters: task_parameter/typeofwork, update Integer, and final_result. Can be passed with (Void, Void, Void)
  • doBackground(task_parameter) { do your work here. the only non optional method}
  • onProgressUpdate(Integer progress) {optional, updating the progress}
  • onPostExecute(final_result) {}
  • You have to have main thread (Handler/Looper)
  • Don’t use AsyncTask in a library
  • Don’t use AsyncTasks for longish tasks that are important/critical. They may get killed by GC when the activity is closed by the user.

For critical functions use: android.app.IntentService (Android 1.5 and later):

“IntentService is a base class for Services that handle asynchronous requests (expressed as Intents) on demand. Clients send requests through startService(Intent) calls; the service is started as needed, handles each Intent in turn using a worker thread, and stops itself when it runs out of work.

This work-queue-processor pattern is commonly used to offload tasks from an application’s main thread. The IntentService class exists to simplify this pattern and take care of the mechanics. To us it, extend IntentService and implement onHandleIntent(Intent). IntentService will receive the Intents, launch a worker thread, and stop th eservice as appropriate.”

Benefits of the IntentService pattern:

  • your activity’s process, while processing that Intent, now has a Service running
  • the Android process killer will try really hard not to kill you
  • but once you’re done handling your piece of work, you’re now properly disposable again
  • very easy way to use a Service

Tips:

  • disable UI elements immediately, before kicking off your AsyncTask to finish the task
  • do some animation (psinner in title bar)
  • progress dialog (for longish operations. use sparingly, can be jarring/interrupting)
  • best of both worlds: start with disabling UI, animating something, start timer, show progressDialog if it’s taking over 200ms, in AsyncTask onPostExecute cancel alarm timer

Profiling your app

Traceview tool

  • toggle in code: dalvik.system.VMDebug#{start,stop}MethodTracing()
  • toggle at runtime: adb shell am profile <process> start|stop <FILE>
  • adb shell getprop (system/build in-memory properties)
  • /data/local.prop overwrite the system-wide properties (phone reboot required)
  • setprop propname value, eg. setprop db.db_operation.threshold_ms 10

print here logging

  • non-public class android.os.PerformanceCollector
  • tracepoint name (start, stop)
  • android.osSystemClock#uptimeMillis()
  • dalvik.system.VMDebug#threadCpuTimeNanos()

New Froyo 2.2 instrumentation

Not a generic debugging feature yet, but almost there

Log vs EventLog

  • Log: text ring-buffer (adb logcat), apps use this one.
  • EventLog: binary structured ring-buffer (adb logcat -b events), intended for low-level platform development work only

EventLog records: db_sample, binder_sample, content_query_sample, content_update_sample, dvm_lock_sample

Instrumentation:

  • Database queries (see system property db.db_operation.threshold_ms)
  • RPC (Binder) calls,
  • ContentResolver,
  • Mutex lock contention in Dalvik.

Other “tricks”:

  • flashing the device cashes, use: echo 3 > /proc/sys/vm/drop_caches
  • change of the phone orientation is equivalent by default to a configuration change.
  • onRetainNonConfigurationInstance()  and   getLastNonConfigurationInstance()