Changing Business Conditions Affect Data Requirements.
Business conditions frequently change that force organizations to analyze business systems to plan for critical changes to stay competitive and improve corporate performance. We are in an era of frequent changes as a result of mergers and acquisitions, restructuring of disparate systems, implementation of new applications, changes in the regulatory framework, and updates to the data comprising the database. When implementing the changes to business systems, efficient data conversion is vital to ensure the quality and reliability of the data.
Understanding of the hidden challenges of a data conversion and carefully managing the process is more likely to deliver reliable data that supports the needs of the business and mitigates the risk of delays and budget overruns.
Planning the Data Conversion
Data conversion refers to the transformation and transfer of data between different systems when replacing the systems or updating them. When converting existing data to a new target application, it can become apparent that the source contains inaccuracies, unknowns, and redundant and duplicate data. Mistakenly, data conversion planning is undervalued and is often seen as a simple matter of shifting data from one system to another. Planning is left until too late and the required resources and the difficulty of the conversion are frequently underestimated. The complexity of data conversion, however, requires thorough and proper planning and a requirements analysis to avoid budget and schedule overruns.
- What kind of data needs to be converted?
- What is the volume of data to convert?
- What is the quality of data?
- Which data should be moved to the new database?
- Which data should not be moved? Should it be purged or archived?
- What is the original data format and what is the final format?
- What are the data conversion standards to use?
- What is the timeline for the project?
- What is the permissible downtime when moving to production?
- What contingency plan is there for the unanticipated?
- Are there any constraints or limitations that will affect data conversion?
- What is the backup strategy and what is the restore process if necessary?
The plan should have the approval or buy-in of the management team, the project team, and the business users.
Involve Business Users Too!
A data conversion project is often a technically focused conversion and assumes that the conversion team knows the existing systems and has schemas for the target application. The focus is on preparing detailed mapping specifications or rules for converting source data to the target.
Those in the business side of the organization who will work with the new system are unlikely to understand the criticality of the data conversion, the IT or data quality issues, and how it impacts their processes. Therefore, it is important to help them understand the data conversion effort and how the quality of data conversion can affect their efforts that make use of this data. Understanding the actual content of the data results in the reduction of significant errors and a high rework rate. Overlooking the data content will result in a target system that will not perform effectively and workarounds will need to be implemented.
Getting the right mix of business and IT at the right levels across the project is important. The data conversion process is a task critical from both business and technical perspectives.
The Process is Critical
Conversion of data from an existing system is one of the most challenging and critical tasks. Mapping data to a new system requires fully understanding the data sources before starting to convert and knowing the destination system in depth. Converting the data into an appropriate format that fits the destination database should ensure:
- Data is converted and transferred correctly.
- Data works in the new destination database.
- Data retains its quality.
- Data consistency is maintained at all times across all systems using that particular data.
Profiling the data prior to the conversion ensures success. Complete visibility of all source data allows the team to identify and address potential problems and will relegate uncertainty and risk, lower costs of iterations and rewriting code, decrease delays and wasted time, and reduce unexpected costs while improving the ability to complete the effort on time and under budget.
Prior to the data conversion, identify the types of data quality problems that may occur such as alpha characters in dates and numbers, orphan records, or multiple users for a single field. Discuss the strategy to ensure data quality and describe the approach to any necessary data scrubbing. Identify outdated or irrelevant data and establish rules for purging or archiving.
Describe the controls and methods to validate the completed conversion to ensure that all data intended for conversion have been converted. Describe the process for data error detection and correction, and the process for resolving anomalies.
Armed with a thorough understanding of the data to convert, the team can configure the rules to accurately map the designated source data to the target. The Rules govern the relationships between data objects and the types of operations permitted. The ideal solution for this is an integrated tool incorporating transformation, cleansing, and matching functions and incorporates capabilities for configuring a rule that contain operations ranging from ‘select’ to ‘select and copy’, archive, delete and transform.
The accuracy and completeness of the data in the target system must be assessed but testing should begin prior to the data conversion and should include several techniques to be thorough.
- Pre-conversion testing
- Post-conversion testing
- Production Conversion
Pre-conversion testing – Identify a data subset for testing. This will verify that the scope of the data conversion and the process is understood, including the data scheme and data cleansing,
During the conversion process, the best solution allows the user to view the data conversion process as it occurs, reporting any exceptions to the data conversion process. The exceptions can be considered and dealt with during the process.
Post-conversion testing – A user should log in to the source system and the target system to compare the information in each system. A user or analyst compares reports from the source system with reports containing the same records prepared from the target system. The tester should build test scripts that provide exact detail of data usage. This is invaluable for confirming that the data elements were mapped to the correct fields.
Acceptance – Having performed due diligence and with the satisfaction the data conversion completed successfully, the team acknowledges acceptance and schedules the move to production.
Production Conversion – The team performs one last test to ensure recent data created is converted without error and that the production data is ready to go.
The team performing the testing should have the goal of minimizing the time required to conduct the validation. It is the final step before moving to production. During the time between the final extract of data from the source system to the implementation of the new system to production, transaction will continue to be processed. These transactions must then be re-entered into the new system.
Data conversion is complex and necessitates specific, careful testing at all stages of the process. Not doing so can result in faulty conversion, corrupt data or poor user experience, which will be damaging to the system’s reputation and its usability as well. A quality and thorough test plan reduces uncertainty and risk.
It is imperative to plan a successful data conversion well with an organized process and thorough testing. Otherwise, there is a high risk the project will go over budget, exceed the allotted time, or even fail completely.
Following a structured methodology will reduce the pain of managing a complex data conversion, but the correct choice of technologies will go a long way to promote a successful outcome. The model solution is a software tool that supports the entire data conversion effort, from profiling and auditing the source(s) through transformation, cleansing, and mapping to the target. The solution should be flexible and highly scalable and require minimal technical expertise.