You need to use the "javascript:" prefix and return an encoded query as a String.

For example, if you have a Company field, then an Employee field, but you only want to let the User select people that are employees of the selected Company, then you would set the following as the Reference qualifier on the Employee field:

javascript: 'company=' + current.company;

That is, we want to look up Users where the company field (on the sys_user table) is equivalent to the company field on the current record.