Data mining Online Test Series 1, Data mining Question and Answers

Data mining is the computational process of discovering patterns in large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics, and database systems, data warehouses, file systems. It is an interdisciplinary subfield of computer science. The overall goal of the data mining process is to extract information from a data set and transform it into an understandable structure for further use.

In No-coupling scheme, the data mining system does not utilize any of the database or data warehouse functions. It fetches the data from a particular source and processes that data using some data mining algorithms. The data mining result is stored in another file.

In loose coupling, the data mining system uses the database or data warehouse for data retrieval. In loose coupling data mining architecture, the data mining system retrieves data from the database or data warehouse, processes data using data mining algorithms and stores the result in those systems. This architecture is mainly for memory-based data mining system that does not require high scalability and high performance.

In tight-coupling, database or data warehouse is treated as an information retrieval component of data mining system using integration. All the features of database or data warehouse are used to perform data mining tasks. This architecture provides system scalability, high performance, and integrated information. There are three tiers in the tight-coupling data mining architecture:
1. Data layer: This layer is an interface for all data sources. Data mining results are stored in data layer so it can be presented to end-user in the form of reports or another kind of visualization.

2. Application layer: It is used to retrieve data from the database. Some transformation routine can be performed here to transform data into the desired format. Then data is processed using various data mining algorithms.

3. Front-end layer: It provides intuitive and friendly user interface for end-user to interact with data mining system. Data mining result presented in visualization form to the user in the front-end layer.

Classification is a classic data mining technique based on machine learning. Basically, classification is used to classify each item in a set of data into one of a predefined set of classes or groups. Classification method makes use of mathematical techniques such as decision trees, linear programming, neural network and statistics.

The prediction, as its name implied, is one of a data mining techniques that discovers the relationship between independent variables and relationship between dependent and independent variables. For instance, the prediction analysis technique can be used in the sale to predict profit for the future if we consider the sale is an independent variable, profit could be a dependent variable. Then based on the historical sale and profit data, we can draw a fitted regression curve that is used for profit prediction.

Genetic Algorithm (GA) is a search-based optimization technique based on the principles of Genetics and Natural Selection. It is frequently used to find optimal or near-optimal solutions to difficult problems which otherwise would take a lifetime to solve. It is frequently used to solve optimization problems, in research, and in machine learning.

Enterprise-wide applications generally range in size from 10 gigabytes to over 11 terabytes.

Data mining consists of five major elements:
1. Extract, transform, and load transaction data onto the data warehouse system.
2. Store and manage the data in a multidimensional database system.
3. Provide data access to business analysts and information technology professionals.
4. Analyze the data by application software.
5. Present the data in a useful format, such as a graph or table.

MPP stands for massively parallel processing database. An MPP database is a database that is optimized to be processed in parallel for many operations to be performed by many processing units at a time.

One of the key issues raised by data mining technology is not a business or technological one, but a social one. It is the issue of individual privacy. Data mining makes it possible to analyze routine business transactions and glean a significant amount of information about individuals buying habits and preferences. Another issue is that of data integrity. Clearly, data analysis can only be as good as the data that is being analyzed. A key implementation challenge is integrating, conflicting or redundant data from different sources.

Decision trees represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID) . CART and CHAID are decision tree techniques used for classification of a dataset. They provide a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. CART segments a dataset by creating 2-way splits while CHAID segments using chi square tests to create multi-way splits. CART typically requires less data preparation than CHAID.

Cluster analysis groups data objects based only on information found in the data that describes the objects and their relationships. The goal is that the objects within a group be similar to one another and different from the objects in other groups.

The final step of knowledge discovery from data is to verify that the patterns produced by the data mining algorithms occur in the wider data set. Not all patterns found by the data mining algorithms are necessarily valid. It is common for the data mining algorithms to find patterns in the training set which are not present in the general data set. This is called overfitting. To overcome this, the evaluation uses a test set of data on which the data mining algorithm was not trained. The learned patterns are applied to this test set, and the resulting output is compared to the desired output.

Data mining is the analysis step of the “knowledge discovery in databases” process, or KDD. The Knowledge Discovery in Databases (KDD) process is commonly defined with the stages:
1) Selection
2) Pre-processing
3) Transformation
4) Data Mining
5) Interpretation/Evaluation.

The First International Conference on Knowledge Discovery and Data Mining (KDD-95) was held August 20–21, 1995, Montreal, Quebec, Canada. Knowledge Discovery in Databases (KDD) and Data Mining are areas of common interest to researchers in machine learning, machine discovery, statistics, intelligent databases, knowledge acquisition, data visualization, high performance computing, and expert systems.

Key challenges faced distributed data mining algorithms include:
1. To reduce the amount of communication needed to perform the distributed computation.
2. To effectively consolidate the data mining results obtained from multiple sources.
3. To address data security issues.

Techniques from high performance (parallel) computing are often important in addressing the massive size of some data sets.