configured by the user. The lower limit of the allowed domain is called />min- For some instruments, there is no limit (or pm{n = -oo). The choice of the lower limit is rather obvious for most instruments. A list of important examples is presented here:

Prices. Genuine asset prices of whatever kind, including FX and equity prices, are never negative. This means: \ = 0.

FX forward premiums/discounts. As explained in Section 2.3.2, the "forward points" can be positive or negative. There is no lower limit (/ = -oo).

Interest rates. These can be slightly negative in extreme cases such as the JPY case discussed in Section 2.3.1 (but these negative interest rates were above -1%). Some theories rely on interest rates staying always positive, but a filter is not allowed to reject slightly negative interest rates if these are posted by reasonable contributors. The filter should use a moderately negative value of pm\n here, e.g. -5%.

Short-term interest-rate futures. These can normally be handled as ordinary prices (where pm\n = 0), but are in fact defined by Equation 2.3 to have no lower limit (but an upper one). In practice, futures prices are quite far from 0, so it does not matter whether we assume a lower limit of 0 or none.

The choice of the lower limit is important for the further treatment. The following errors lead to complete invalidity:

Quotes that violate the monotonic sequence of time stamps (i.e., quotes with a time stamp before the previously treated quote). In some software environments, this is an impossible error.

A domain error. An illegal level p of the filtered variable (i.e., p < /?mm) as opposed to a merely implausible level.

Invalid scalar quotes with an error of this kind do not enter a scalar filtering window and are completely ignored in all further filtering steps. We mark them by setting d = 0. This is a fundamentally stronger statement than merely giving a very low credibility as a result of the scalar filtering window.

In the case of bid-ask quotes, the three resulting scalar quotes are tested individually:

Bid quote. Domain error if bid quote pbid < / -

Ask quote. Domain error if ask quote pask < / -

Bid-ask spread. Domain error if & < pbid-

Thus it is possible that the same quote leads to a valid bid quote passed to the scalar filtering window of bid prices and an invalid ask quote that is rejected.

The domain test of bid-ask spreads needs to be further discussed. First, we might interpret bad values (/>ask < Pb\d) as the result of a sequence error. In other words, if the contributor typed ask-bid instead of bid-ask, this would be an error that could be corrected by the filter. This interpretation, although being true in

many cases, is dangerous as a general rule. We prefer to reject all ask quotes that are less than the bid quote.

On the other hand, a more rigid test might also reject zero spreads. However, there are some quote contributors to minor markets interested only in either bid or ask or middle quotes. These contributors often produce formal quotes with = pask- In some markets, such quotes are the rule rather than the exception. A filter that rejects all of those quotes is throwing away some valuable and irreplaceable information.

The solution looks as follows. First, there is a filtering option of generally rejecting zero spreads (i.e., the case pbid = Pask)- if the user chooses this option, the quote splitting algorithm will act accordingly. Otherwise, zero spreads can be accepted, but they have low credibilities in a market dominated by positive spreads. This is further explained in the next section.

4.6.3 Transforming the Filtered Variable

The filtered variable is mathematically transformed in order to reach two goals:

1. A simpler (e.g., more symmetric) density function. The basic filtering operations (e.g., Equation 4.6), assume a roughly symmetric distribution of the scalar quote values (and their changes). Some variables, mainly the bid-ask spread, have a skewed distribution. The filtering method contains no full-fledged analysis to determine the exact nature of the distribution. This would be too much for an efficient filter algorithm. The idea of the transformation is that the mathematically transformed variable has a more symmetric distribution than the raw form. For the logarithm of bid-ask spreads, this has been demonstrated in Muller and Sgier (1992).

2. The transformed variable should not depend on units such as Japanese Yens per U.S. Dollar. Relative changes have the advantage of being comparable between different financial instruments and different time periods of the same instrument. They do not depend on the units in which the different rates are expressed. A usual way to work with relative, unit-free variables is to take the logarithm of the raw variables such as prices.

The rules of the mathematical transformation are closely related to the validity tests of Section 4.6.2. The transformation never fails because all illegal quotes have already been removed by the domain tests. The transformed quote value is denoted by x and used in many formulas of Section 4.4.

For single-valued quotes, bid quotes, and ask quotes, the following transformation is made:

x I loS(P ~ Pmin) if Pmin > -oo exists I p otherwise

For bid-ask spreads, the transformation is

(4.44)

x = Spread = 45.564 Vask ~ *bid

(4.45)

where Xbid and xa.sk are results from Equation 4.44. Equation 4.45 has been chosen to return a value similar to log(xask - -*"bid) -fconstant for a wide range of arguments *ask - *bid of typically occurring sizes. Indeed, a logarithmic transformation of spread values would be a natural choice. The reason to use Equation 4.45 rather than a logarithmic transformation is related to zero spreads.2 A logarithmic transformation would make zero spreads impossible (as log(O) = -oo). When inserting a zero spread in Equation 4.45, we obtain the legal result x = 0. This value is far away from typical ranges of values obtained for positive spreads, so its credibility is likely to be low in normal situations. When zero spreads become a usual event, the filter will start to accept them.

4.7 UNIVARIATE FILTERING

Univariate filtering is the top level of the filter. All the main filtering functions are managed here. The full-quote filtering window with its quote splitting algorithm of Section 4.6.1 is on a lower hierarchy level (see Table 4.1). Thus the univariate filter sees full quotes before they are split; it has access to all components of a full quote in their raw form (with no transformation). The tasks of univariate filtering are as follows:

Serving as the main configuration of a filter.

Analyzing those data errors that affect not only individual quotes but a whole continuous sequence of quotes. The presence (or absence) of such a general error defines the filtering hypothesis. Two such cases were found in financial data and are therefore covered by the filter:

1. Decimal errors. A wrong decimal digit of the quote, corresponding to a constant offset from the true quote.

2. Scaling factor. The quote deviates from the true level by a constant factor, often a power of 10.

Both cases are further discussed here.

Creating a new full-quote filtering window for a newly detected filtering hypothesis.

Managing filtering hypotheses and their full-quote filtering windows during their lifetimes, selecting the winning hypothesis.

In the case of an error hypothesis, correcting the error of new incoming quotes according to the hypothesis and passing the corrected quotes to the full-quote filtering window.

Packaging the filtering results to be accessed by the user.

Recommending a suitable build-up period of the filter prior to the desired start date of the filtering result production, based on the filter configuration. Typical sizes are from weeks to months.

The treatment of zero spreads is discussed at the end of Section 4.6.2.