There’s a right way and a wrong way to keep test data out of production. Citigroup chose the wrong way:
It turned out that the error was a result of how the company introduced new alphanumeric branch codes.
When the system was introduced in the mid-1990s, the program code filtered out any transactions that were given three-digit branch codes from 089 to 100 and used those prefixes for testing purposes.
But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC.
The SEC routinely sends requests to financial institutions asking them to send all details on transactions between specific dates as a way of checking that nothing untoward is going on. The coding error had resulted in Citigroup failing to send information on 26,810 transactions in over 2,300 such requests.
Citigroup was fined $7,000,000 for the problem which probably stemmed from a lack of communication.