“A Framework for Valuing the Quality of Customer Information”


PhD Proposal II

Draft 3


Greg Hill


  1. Summary


This is a proposal for a PhD research project. The objective is to develop a method that Information Systems practitioners can employ to help them invest economically in information quality for Customer Relationship Management processes. Starting with a theory of the value of classifying customers, the proposal is to describe a quantitative model for CRM processes that supports an “investment view” of activities related to information quality. The intent is for this model to be the basis for a method used by analysts and decision-makers to select, justify and implement organisational investments in information quality.


The research proceeds in two phases: firstly, method development with a conceptual study and semi-structured interviews with practitioners; secondly, method validation with a field trial of the method, followed by a focus group evaluation.


The research develops and extends ideas from the sub-disciplines of Information Quality and CRM in a way that allows IS practitioners to plan and evaluate the economic value of their initiatives.


  1. Introduction


This research is concerned with information management, specifically methods for valuing investments in Information Quality for Customer Relationship Management processes. Such methods are important because they assist decision-makers to compare competing proposals (selection), explain to others the reasons for decisions (justification), and form agreements with other parties (implementation).


The domain of application is the subset of Customer Relationship Management (CRM) processes that are based on customer classification: organisational processes that allocate each customer into one of a small number of treatments. This research will examine these processes in the context of targeted (or direct) marketing activities in large retail service organisations.


The research seeks to contribute to Information Systems (IS) sub-disciplines of IQ and CRM by drawing in concepts from the reference disciplines of Finance (for quantifying financial risks and benefits) and Machine Learning (for quantifying performance of customer classification). The innovative aspect is the synthesis of these concepts into a validated method for partially valuing IQ activities in some CRM processes.


This document is organised as follows. The next section provides background on IQ and CRM within the IS literature, and valuation methods and classifier performance from their respective reference disciplines (Finance and Machine Learning). The third section articulates the research questions and the form of the answers the proposed research provides. The fourth section explains the contribution to IS academia and practice, as well as highlighting the limitations of the research. The fifth section outlines the design of the research, based on a Systems Development approach. The final section contains a project plan for completing the research within two years.

  1. Background


This research is an examination of the valuation of Information Quality (IQ) for Customer Relationship Management (CRM). The key idea is that some CRM processes are essentially about classifying customers, and that when customers are misclassified costs of different kinds are introduced. One reason an organisation may undertake IQ activities is to improve this classification. This raises the question: where should they focus their resources (selection)? How do they communicate this to others (justification)? How can they form agreements (implementation)?


It is proposed here that practitioners may answer these questions by applying a sound method for valuing the quality of information in their context. The development and validation of this method is the goal of this research, and it is based on widely accepted principles and measurement constructs from the fields of Information Economics (based on Utility Theory) (Lawrence, 1999) and Classifier Performance (based on Information Theory) (Hand, 1997). The term “Information Quality” is understood through the Semiotic framework (Shanks and Darke, 1998), and this is the basis for describing organisational IQ activities.



  1. Information Valuation Methods


It is proposed that a decision-theoretic view of information valuation be taken. That is, the value of information is the expected marginal utility of a reduction in uncertainty for the decision-maker. In other words, information is a measure of the extent to which a decision-maker’s uncertainty about the world is reduced (Shannon and Weaver, 1949). If this reduction in uncertainty leads to them taking a different course of action, then the difference in utility (well-being) of this new state of affairs over what would have been is the marginal utility (or pay-off). If, in advance, the decision-maker expects this pay-off to exceed the costs, then the information is valuable to them. This is a conventional approach used as the basis for much of the vast literature in Information Economics (Lawrence, 1999).


However, this approach is out of step with some Information Systems authors tackling this subject. For example, Moody and Walsh cite numerous researchers that stress the importance for organisations to value information (Moody and Walsh, 1999). While they rightly argue for the benefits of regarding information as an organisational asset, rather than as an expense (as at present), they urge the use of historical cost valuation over utility valuations. While acknowledging that theoretically utility is to be preferred, the difficulties of identifying future cashflows and the lack of auditability mean that it is not suitable for use on the balance sheet.


For use in business cases and other forward-looking assessments, historical cost is problematic, relying as it must on reference to (similar) past projects. This research project seeks to show how the utility valuation approach can be revived by identifying the future cashflows for a small but significant class of situations: CRM processes. In this research, Net Present Value (NPV) is the financial measure for determining the value of future cashflows, subsuming related constructs such as Internal Rate of Return, Return on Investment and Payback Period, amongst others (this is discussed in any standard text on Capital Budgeting. See for example, P.V. Viswanath’s online tutorial at http://webpage.pace.edu/pviswanath/class/320/notes/invrules.html)


Additionally, Moody and Walsh overlook the role that Information Theory plays in providing a theoretical basis for the valuation task. Specifically, by failing to distinguish between data (symbols) and information (uncertainty), the notions of “accuracy” “redundancy” and so on described in their “Laws of Information” are left unmeasurable, rendering their stated goal unachievable. Shannon’s formulation of Information Theory – rather than merely describing bit rates of data on the wire – is in fact the basis of a ready-made framework for quantifying these very constructs (Kononenko and Bratko, 1991). This research project seeks to apply this widely-used and carefully-formed framework for measuring “accuracy” in the context of CRM processes.


  1. Information Quality


Information Quality is an IS research area that seeks to apply modern quality management theories and practices to organisational data and systems. This involves building and applying conceptual frameworks and operational measures for understanding the causes and effects of Information Quality problems.


A number of proposals have been made in this area, for example Wand and Wang have an ontologically-based framework consisting of four intrinsic dimensions: complete, unambiguous, meaningful and correct (Wand and Wang, 1996). Dozens of IQ frameworks and variants, each with several dimensions comprised of scores of attributes, have been identified by academics and practitioners (Lee et al, 1999, has a comprehensive review).


These frameworks are very general, and are intended to apply to all types and uses of organisational data or information. While important for furthering organisational understanding, they have a limited ability to influence decision-making as they do not address directly the notion of value (Moody and Walsh, 1999). Also, they do not address directly concepts of rule quality (Dean et al. 1996). In contrast, this proposal is concerned with developing measures that relate the effects of initiatives to the classification of customers within organisational processes.


The Information Quality academic discipline places emphasis on conceptual frameworks and subjective measures, for example the AIMQ methodology developed with MIT’s TDQM program (Lee et al 1999). However, at a Data Quality workshop hosted by the National Institutes for Statistical Sciences in 2001, one of the key recommendations was that “Metrics for data quality are necessary that … represent the impact of data quality, in either economic or other terms” (NISS, 2000). This is very difficult owing to the very broad impacts of data quality within – and beyond - an organisation, and the large range of purposes for which particular data are used. A final confounding factor is the diffused intangibility of many of these impacts.


Efforts at defining and measuring objective measures of IQ – though less widely employed – have been made. For example, Kaomea (1994) applied a decision-theoretic analysis involving probabilities and pay-offs to argue for a method of valuing data content in context. A methodology for developing IQ metrics known as InfoQual has been proposed (Dvir et al. 1996), while the Data Quality Engineering Framework has a similar objective (Willshire et al. 1997). These efforts focus on measuring properties of data (possibly complementing subjective user ratings), rather than process outcomes. Also, the very general nature of the situations these proposals address means they offer little support for the valuing task, as shown by the NISS call for economic measures of data quality.


Since the research is concerned with the value of information quality activities within organisational processes, a framework is required to describe and group these activities. This project will use the semiotic framework as it provides a theoretically sound and comprehensive framework derived from the field of semiotics, or semiology (Shanks and Darke, 1998). Under this framework, information quality goals are grouped into four abstract levels that build upon each other:



The focus is on the pragmatics layer. This is because it is in the use of information that value (utility) is realised, and this layer explicitly deals with use, subsuming meaning and form, in the pragmatic level. Shanks and Darke (1998) define pragmatics as the usefulness and useability of the symbols:


“Usability is the degree to which each stakeholder is able to effectively access and use the symbols. Usefulness is the degree to which stakeholders are supported by the symbols in accomplishing their tasks within the social context of the organisation. Desirable characteristics relating to pragmatic data quality include timeliness, understandability, conciseness, accessibility and reputation of the data source.”


By restricting ourselves to the context of CRM processes, then the task is one of customer classification and the user of the symbols (stakeholder) is an organisational process. This task-centric definition of usefulness as “degree of support” fits with the commonly-accepted view within the Machine Learning community of information being useful (as opposed to misleading) when the decision-maker’s uncertainty is reduced in accordance with true belief (Kononenko and Bratko, 1991).





  1. Customer Relationship Management


There has been considerable academic interest in Customer Relationship Management (CRM) strategies, applications and processes, with some 600 papers published in the last five years (Romano 2001). While quality data (or information) about customers is identified as key to the success of CRM initiatives it is not clear exactly how one should value this. Indeed, even the real costs of poor customer data are difficult to gauge due to the complexities of tracing causes through to effects. This is part of the much larger data quality problem. At the large scale, The DataWarehousing Institute estimated that – broadly defined - poor data quality costs the US economy over $US600 billion per annum (TDWI, 2002).


Following Meltzer (2002), a CRM process is seen an organisational process for managing customers. He identifies six basic functions:



At the core of these processes is the idea of customer classification: a large set of customers is partitioned into a small number of target sets. Each customer in a target set is treated the same by the organisation, though each may respond differently to such treatments. This approach seeks to balance the competing goals of effectiveness (through personalised interaction with the customer) and efficiency (through standardisation and economies of scale).


Customer/Treatment Allocation


a priori customers classifier a posteriori customers


Treatment A


Treatment B



uncertainty


Figure 1: Using a Classifier to Map Customers to Treatments Under Uncertainty


For example, a direct mail process might require partitioning a customer list into those who are to receive the offer, and those excluded. In this case, there are four possible outcomes from the treatment dimension “Offer/Not Offer” and the response dimension “Accept/Not Accept”. The objective of the classifier is to correctly assign all customers to their correct treatment (ie accepting customers to “offer”, not accepting customers to “Not Offer”).


A misclassification will result in costs (either direct or to revenue) of different magnitudes. In order to apply the utility valuation approach, the future cashflows for the outcomes must be ascertained. Rather than an exhaustive examination of these cashflows, it is proposed here to use the change in customer value measure as the pay-off.


Customer Value is sometimes called Lifetime Value (LTV) or Customer Lifetime Value (CLV) or Future Customer Value. It is widely used as the basis for evaluating CRM and Database Marketing initiatives, and is now identified as a standard by the Database Marketing Institute (Hughes 2002). The idea is that the worth of a customer relationship to an organisation can be evaluated by adding up the revenues and costs associated with servicing that customer over the lifetime of the relationship, taking into account future behaviours (such as churn) and the time value of money (Berger et al. 1998). As such, it represents the Net Present Value of the customer relationship.


So, for the direct mail example, it may to tempting to value a “false negative” (ie failing to make an offer to a customer who will accept) as the lost earnings from the particular sale. However, the real cost is likely to be larger as the risk of the customer churning (swapping to a different provider), or abandoning the service all together, plus costs of “Re-Activation”, must be priced in. Further, it will depend on the customer themselves: are they high-spending, or new to the organisation, or “locked-in” via contract? Such modelling is the bread-and-butter of many marketing analysts. In many cases, a business-owner will provide a “soft estimate” of these values for decision-making purposes.


It is posited that the Customer Value measurement is the most suitable economic measure for describing the impact of information quality. This allows for a mixture of subjective and objective value, as deemed necessary by the decision-maker. Hence, it is not required to model all the financial implications of information quality, just enough to satisfy decision-makers for the purposes at hand.


  1. Classifier Performance Measurement


The analysis of CRM processes as classifiers with pay-offs is not widely understood in the Information Systems literature. Yet, the “de-coupling” of classifier performance and the value of the outcomes has been advocated in the statistical and machine learning literature (Ming 2002). This is to allow comparison of different classifiers in the same task, and prediction of classifier performance in context in advance of its deployment (Piatetsky-Shapiro et al. 1999). To that end, this discipline has formulated models and measures of performance that can be adapted for predicting and describing classifier performance in CRM processes.


There are two broad categories of measures identified within the literature. The first examines ratios of “true positives” (eg “hits” in direct marketing ) and “false positives” (eg. “misses”). This concept is addressed generically by the ROC concept (Ming 2002), which subsumes five earlier measures of classification performance: Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value and Efficiency (or accuracy) (Kononenko and Bratko, 1991). A marketing-specific treatment is found in the L-Quality metric proposed by Piatestky-Shapiro et al. (2000), based on earlier work in direct marketing (Piatestky-Shapiro et al. 1999).


The second category of measures is those derived from Information Theory. These measures relate to entropy, or the reduction of uncertainty, first proposed by Shannon (1948). One approach widely used within the machine learning literature is that described by Kononeko et al. (1991). The “average information score” and “relative information score” measure how much uncertainty is reduced by a classifier, on average.


While these are sophisticated measures describing the performance of the classifier (CRM process), they do not take into account the consequences of this performance. As such, there appears to be agreement within the literature that misclassification costs should be used (Hand, 1997) for evaluation. This is calculated by multiplying of the pay-off of each outcome with its frequency. So for a two-treatment classifier like the direct mail example, there are four outcomes. As discussed above, these costs should be changes to customer value, rather than simply one-off earnings.


This approach to valuing the performance of CRM processes is sufficiently generic to characterise a wide range of CRM processes in general, and the different initiatives under examination. During planning, estimating performance in advance of implementation is required, while measurements based on observable outcomes are used for review. In both cases, the performance measures drive a customer value-based model to derive the financial outcomes (see Figure 2).
















Figure 2: Valuing IQ Activities with Process Performance and Customer Value


  1. Research Questions



  1. How does improving customer classification create value for an organisation?


This question is answered by developing a model that allows an analyst to 1) define customer classifier performance; 2) define value for the organisation; 3) describe a CRM process of interest; and, 4) estimate value as a function of customer classification performance.


  1. How do customer Information Quality activities relate to customer classifier performance?


This question is answered by 1) a definition of IQ activities; 2) cases of IQ activities; 3) evidence of customer classifier performance without the IQ activities; 4) evidence of customer classifier performance with the IQ activities; and, 5) analysis of the cases.


  1. Does this suggest a method for valuing Information Quality activities?


This question is answered by developing an IQ valuation method 1) based around the classifier value model in Question 1; and, 2) drawing on the analysis from Question 2 to examine the benefits and limitations of the proposed method.


  1. Is such a valuation likely to be accepted by decision-makers as superior to that from existing methods?


This question is answered by 1) using the method to undertake an after-the-fact valuation of an IQ activity; 2) comparing this with the valuation from the existing method; 3) presenting both valuations to target decision-makers; 4) gathering evidence of their opinions on the wider acceptance of the proposed method; and, 5) analysing this evidence.


  1. Justification



  1. Contributions to Information Systems Academia


The main contributions to IS academia are expected to be in the quantifying some of the benefits of IQ activities in CRM processes.



These measures and methods may be useful for comparing processes across organisations or industries, or tracking changes over time for one process. As suggested by Figure 2, there are other possible antecedents to improved classification. For example, training of customer service staff, or market research. The valuation method developed here may be adapted to examine these activities, and compare benefits between them.



  1. Contributions to Information Systems Practice


Information Quality is a difficult area for practitioners as it requires large investments, and the benefits are uncertain and defused. Many pundits and industry gurus argue that the key to CRM success is the quality of the information, particularly the integration from multiple sources. As a result, there are many products and services available to organisations to improve the quality of their information as used in CRM processes. Examples include data validation, cleansing, matching, augmenting, scoring and mining. Delivery can be through in-house projects, outsourced service provisioning, “off-the-shelf” vendor solutions, packaged applications or custom development.


Ultimately, an organisation will have to investigate, evaluate, plan and manage these disparate initiatives. Accountable managers are expected to justify their decisions from an investment perspective with financial arguments. Academic research in Information Quality does not have a focus on providing this kind of “investment view” support for practitioners.


This research seeks to fill this gap by developing a method that practitioners can use to articulate with some certainty a “lower bound” on the expected value of IQ activities, in a way likely to be understood and acted upon by decision-makers who control funding.



  1. Boundaries and Limitations


The primary limitation of this research is that it is a partial valuation method: that is, it does not attempt to discover all the value arising from IQ activities (see Figure 2). For example, legal and regulatory compliance and employee morale may not be incorporated in the CV model, yet may still be important goals of IQ activities. There are also wider intangibles arising from IQ activities to do with operational, managerial and strategic benefits that are not considered here.


The rationale for this narrow scope is to provide a compelling analysis of the tangible value of IQ activities, including practicable measures, to encourage investment. As such, it represents a lower bound, and is a conservative investment criterion (ie, likely to result in under-investment rather than over-investment). This is expected to be an improvement on current practice.



  1. Research Design


The goal of the research is a method to be t used by practitioners and researchers to plan investments in Information Systems. As such, a means of describing the method is required. This is supplied by the framework of Avison and Fitzgerald, which recommends using the following seven components (Avison and Fitzgerald, 1995):


        1. Philosophy

          1. Paradigm

          2. Objectives

          3. Domain

          4. Target

        2. Model

        3. Techniques and Tools

        4. Scope

        5. Outputs

        6. Practice

          1. Background

          2. Userbase

          3. Players

        7. Product


Note that this framework applies to “methodologies” rather than “methods”, and there is some debate within the academic community about the use of these terms (see Avison and Fitzgerald, 1995). The term “method” is used here, as a “methodology” addresses the “end-to-end” view of IS projects, rather than just the planning and evaluation parts. That said, this framework still is useful for describing such a method.


The object of the research is an artefact that embodies the method. Consequently, the research is organised around the Systems Development approach following the recommendations of Burstein and Gregor (1999). It should be noted that here “system” refers to an artefact that embodies a process or method for organisational analysis, rather than a physical instantiation of eg software. This approach naturally spans the theory-building and theory-testing aspects of applied research as:


“after the theory is proposed it needs to be tested in the real world to show its validity and recognise its limitations, as well as to make appropriate refinements according to the facts and observations made during its application.” (p 124)


The theory-building will be undertaken with a conceptual study and semi-structured interviews; while the theory-testing will employ field trials and a focus group. Both are empirical and involve contact with industry practice. Some authors, such as Burstein and Gregor (1999), suggest that the System Development approach is a form of Action Research. However, it is argued that this case is not Action Research. Kock et al (Kock, McQueen and Scott, 1997) propose a test for Action Research as being that where


“intervention [is] carried out in a way that may be beneficial to the organisation participating in the research study”


Since we are not concerned with actually intervening in the organisation during this research, it should not be considered Action Research. Further, since there is no objective of implementing the method within the organisation, there is no imperative to trace the impact of the changes throughout the organisation – another aspect of Action Research (Burstein and Gregor, 1999).


This may seem contradictory – a field trial after all requires a realistic situation if it is to be more than an experiment. The key is that the situation under examination has been dealt with already (after-the-fact analysis). Deploying the method within an organisation and assessing the impact is out of scope, and should only be undertaken if this research is fruitful.







Field Trial

Focus Group









Method Development
Method Validation



Figure 3: Empirical Phases within Systems Development Approach


  1. Method Development


This phase seeks to develop an artefact that embodies the method. This artefact will consist of documentation, diagrams and spreadsheets, while the method will be described in terms of the seven components identified above. This will be achieved by a concurrent mixture of a) an ongoing conceptual study, and b) a series of semi-structured interviews.


The conceptual study will seek to provide evidence to help answer Research Question 1 (analytical relationship between classification performance and customer value). Literature sources will be academic papers and conference proceedings from information systems and CRM-related disciplines (marketing) and industry white papers, briefings and seminar papers from organisations concerned with information quality and CRM. They will be sourced, analysed and cited in accordance with academic research practices.


The interviews seek to provide evidence to answer Research Question 2 (empirical relationship between IQ activities and CRM processes). The discussion focus on practitioner views on planning and evaluation measures for IQ activities in the CRM domain. Subjects will be drawn from the professional networks of the researcher, supervisor and sponsor. They will be practitioners with expertise in the areas of CRM, IQ, data mining and IS investment.


The interview will be semi-structured in that there will be a set of topics discussed sequentially, but we are concerned with gaining a deep insight into the subjects’ opinions and direct experience. The interviews will last at least an hour and a half, and will be organised into four parts:



The amount of time spent on any particular topic will vary depending on the level of interest and expertise of each subject. The number of subjects is expected to be four to eight, and the interviews will take place over a six month period. Data collected will include:



Each subject will be offered access to the completed PhD thesis, notification of papers arising from the research, and a special practitioner-oriented summary paper to be written by the researcher within one year of data collection.


Evidence collected from the conceptual study and interviews will be used to develop the method and produce a prototype artefact, which in turn will be the basis for addressing Research Question 3 (feasibility of developing a method).


At the end of this stage, the method will be described in terms of the Avison and Fitzgerald framework, a prototype artefact will be developed and evidence for answering Research Questions 1, 2 and 3 will be accumulated.


  1. Method Validation


The purpose of this stage is to provide some evidence to validate the proposed method, and help answer Research Question 4 (acceptance of the method). Assessing a prototype of Systems Development research is difficult and discussed within the IS research literature. This research will use as a guide the five criteria proposed by Burstein and Gregor (1999) with example questions, taken in part from Miles and Huberman (1994).


  1. Significance

Is there theoretical significance?

Is there practical significance?


  1. Internal Validity

Do the methods work?

Have rival methods been considered?

Has sufficient evidence been collected in evaluating the methods?


  1. External Validity

Are the findings congruent with existing theory?

Can the findings be applied elsewhere?


  1. Objectivity/Confirmability

Are the study’s methods described in detail?

Are the researchers explicit about personal assumptions, values and biases?


  1. Reliability/Dependability/Auditability

Are the research questions clear?

Are basic constructs clearly specified?


In order to gather evidence to help address these criteria, this stage has two empirical phases: the first is to use the prototype artefact to apply the method to realistic situations in field trials; the second is to assess the resulting valuations with a focus group.


The objective of the field trial is to demonstrate that the method – as embodied by the artefact - can be applied to realistic situations, helping answer Research Question 3. This is an important step linking conceptual developments of the earlier stages with the intended use of the method in practice. The deliverable from this phase is a set of method outputs (valuation analyses).


The test analyst is the research candidate, an experienced business analyst, and the target situations are planned to be past internal projects of the Industry Sponsor (Telstra Retail). These projects should address Information Quality for a CRM process, and have had some kind of valuation analysis prepared already (eg. business case or costing). While the number of field trials is contingent upon availability of suitable projects and the scope of the method, two to three instances strikes a balance between demonstrating generalisability and overloading the focus group.


The focus group is to independently assess the validity of the output of the method (the valuation analyses). In particular, the focus group will provide evidence of expert opinion into the acceptability of the valuation produced by the proposed method, over the existing valuation. Also, they will be asked to describe the range of initiatives and organisations for which this method is likely to be useful. Evidence of these opinions is the deliverable from this phase.


The focus group comprises four to six academics and practitioners familiar with the research project and with expertise in the area: as such, interview subjects are likely candidates. At least one member will have an interest in the investment situations from the field trial.


This stage is not intended to be a comprehensive test of the method or artefact: the focus is on validating the outputs of the method with expert opinion, rather than evaluating the usability of the artefact by analysts, or likely future adoption within organisations. This is in accordance with the Systems Development research approach.


  1. Project Plan


The following is a high-level project plan for completing the Ph.D. within two years.


Phase 1 – Framing (12 months)


Activities


Deliverables


Risks


Time - Completed by January 2002


Phase 2 – Method Development (6 months)


Activities


Deliverables


Risks


Time - Completed by August 2003


Phase 3 – Method Validation (6 months)


Activities


Deliverables


Risks


Time - Completed by January 2004


Phase 4 – Analysis and Writing (12 months)


Activities


Risks


Time - Completed by December 2004


  1. References


Avison, D. E., Fitzgerald, G. (1995), “Chapter 7: Methodologies: Issues and Frameworks” in Information Systems Development: Methodologies, Techniques and Tools, Second Edition, McGraw-Hill, Berkshire, UK.


Berger, Paul D. and Nada I. Nasr (1998), “Customer Lifetime Value: Marketing Models and Applications” Journal of Interactive Marketing

Burstein, F. and Gregor, S. (1999) "The Systems Development or Engineering Approach to Research in Information Systems: An Action Research Perspective", Proceedings of the Australasian Conference on Information Systems, Victoria University, Wellington, NZ.  

Dean P., Famili, A.. Comparative Performance of Rule Quality Measures in an Induction System. Applied Intelligence Journal. 1996, ftp://ai.iit.nrc.ca/pub/iit-papers/NRC-39188.pdf

Dvir R., Evans, S. (1996). "A TQM approach to the improvement of Information Quality", in the proceedings of the 1996 conference on Information Quality, MIT, http://web.mit.edu/tdqm/papers/other/evans.html

Hand, D.J. (1997), “Construction and Assessment of Classification Rules”, John Wiley & Sons, West Sussex, England, UK

Hauser, J.R. and D. Clausing (1988), "The House of Quality," Harvard Business Review, May-June, pp. 63--73.

Hughes, A (2002) “How Lifetime Value is Used to Evaluate Customer Relationship Management”, Database Marketing Institute, http://www.dbmarketing.com/articles/Art194.htm

Kaomea, P. (1994), "Valuation of Data Quality: A Decision Analysis Approach," Massachusetts Institute of Technology (MIT) Sloan School of Management, Cambridge, MA, TDQM-94-09, http://web.mit.edu/tdqm/www/papers/94/94-08.html

Kock, N., McQueen, R.J. and Scott, J.L. (1997), Can Action Research be Made More Rigorous in a Positivist Sense? The Contribution of an Iterative Approach, Journal of Systems and Information Technology, V.1, No.1, pp. 1-24.


Kononeko, I. Bratko I. (1991) “Information based evaluation criterion for classifier's performance”, Machine Learning Journal, Vol 6, pp67-80.


Ming, L. (2002) “Brief Report: ROC Analysis in Machine Learning”, University of Bristol, Dept of Computer Science, Technical Report 2002-3-13, www.cs.bris.ac.uk/~ml1513/doc/roc.pdf


Lee, Y., Strong, D., Kahn, B. and Wang, R. (2002) “AIMQ: A Methodology for Information Quality Assessment", Forthcoming in Information & Management, http://web.mit.edu/tdqm/www/tdqmpub/AIMQJun02.pdf


Lawrence, D.B. (1999), “The Economic Value of Information”, Springer-Verlag New York, NY, USA.


Meltzer, M. (2002), “CURARE Drives CRM”, in DM Direct June 2002, http://www.dmreview.com/master.cfm?NavID=55&EdID=5316


Moody, D., Walsh P. (1999) “Measuring the Value of Information: An Asset Valuation Approach”, European Conference on Information Systems 1999, http://www.dama.org.au/melbourne/ValueOfInformation.pdf



NISS, (2000) “Affiliates Workshop on Data Quality”, National Institute of Statistical Sciences, http://www.niss.org/affiliates/dqworkshop/report/dq-report.pdf


Piatetsky-Shapiro, G. and Steingold, S (2000) Measuring Lift Quality in Database Marketing, ACM SIGKDD Explorations, December 2000, http://www.kdnuggets.com/gpspubs/sigkdd-explorations-2000-12-lift-quality.pdf


Piatetsky-Shapiro, G. and Masand, B. (1999) “Estimating campaign benefits and modeling lift”, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, http://www.kdnuggets.com/gpspubs/kdd99-est-ben-lift/


Romano, N. C. (2001) Customer Relationship Management Research: An Assessment of Sub Field Development and Maturity, Proceedings of the 34th Hawaii International Conference on System Sciences


Shanks, G. and Darke, P (1998) Understanding Data Quality in Data Warehousing: a Semiotic Approach, Proc. MIT Conference on Information Quality, I.Chengilar-Smith, L. Pipino (eds), Boston (November), pp 247-264


Shanks, G. (1999), Semiotic Approach to Understanding Representation In Information Systems, Proceedings of the Information Systems Foundations Workshop, 1999. http://www.comp.mq.edu.au/isf99/Shanks.htm


Shannon, C. and Weaver, W. (1949). A Mathematical theory of communication. Univ. of Illinois Press, http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf


TDWI, (2002), “Data Quality and the Bottom Line: Achieving Business Success through a Commitment to High Quality Data”, The Data Warehousing Institute, http://www.dw-institute.com/research/display.asp?id=6064


Wand, Y. and Wang, R. (1996) Anchoring Data Quality Dimensions in Ontological Foundations, Communications of the ACM, 39:11, 86-95


Wang, R. and Strong, D. (1996) "Beyond Accuracy: What Data Quality Means to Data Consumers," Journal of Management Information Systems, Spring 1996, Volume 12, No. 4. pp. 5-33.

Willshire, M. J. M., Donna (1997). "A Process for Improving Data Quality." Data Quality 3(1): 8, http://www.dataquality.com/997meyen.htm


Page 1 of 19