The problem

DIB Bank needed to provide usable test data to effectively test its systems prior to a migration project.

However, due to the sensitive nature of the financial records involved, the bank needed to find a way to de-identify the data while ensuring that the data sets remained meaningful.

At the same time, it didn't make sense to send this xx-sized amount of big data to test environments.

And TDM processes were expected to become living processes and run continuously to meet the need for testing.

The challenge

The data being masked in different ways does not comply with referential integrity, cross-system integrations, or application specific requirements.

Slippage in testing deadlines and go live because of issues with test data refresh and test data availability.

There is very little data available to test compared to voluminous production data, thus hindering test efficiency and quality.

Test data set is of same size as to production data, thus hindering test efficiency and speed.

Team waste significant time preparing data rather than testing the application.

Team depend on another team or centralized team to provide the required test data.

Team complains test data generation is a very laborious and complicated process.

Teams not able to reuse test data set.

Teams not able to regenerate same test data set with precision and accuracy.

There is a huge dependency on the upstream systems to create test data.

Increases in data set size, upstream systems, database instance and data sets makes it difficult to manage the test data.

Performance tests require data to be either production scale or representative of production distributions.

It is a known fact that automation requires highly stable, predictable data sets compared to manual testing that can easily adapt to a higher degree of variability.

The solution

Broadcom Test Data Manager (PII) TDM Discovery and Profiling to address data privacy and compliance issues related to compliance regulations and your organizational mandates.

It allowed us to identify across multiple data sources and provided our roadmap for masking and generating synthetic data.

First of all, 20 billion rows of data in approximately 1400+ tables have been consistently masked in order to continue the processes on the test side and keep up with the release schedule.

Then we moved on to Subset creation, Masking and synthetic data generation steps through best practices, which is the method for 2-week sprints in the Agile process.

We provided each tester with their own copy of the test data.

We provided users with the Portal infrastructure that will instantly make data available to testers with a minimum footprint.

Thanks to the synthetic test data we produced according to business rules, we met the test data needs of the users in the fastest time.

Thanks to the integration with MF ALM product, it was ensured that test users can access the reliable data they need through MF ALM.

After the deployment, the relevant test data was generated within the DevOps Pipeline.