Big Data Security in the tech industry

Initial Situation

Private and enterprise customers of telecommunications providers today expect tariffs that are as cost-efficient as possible and tailored to their individual needs. To offer its existing customers tariff adjustments based on a business intelligence (BI) analysis and to capture and assess individual customer performance, the tech company decided to implement advanced Big Data analytics and machine learning processes. The goal: to create a foundation for analyses that deliver benefits for both the customers and the provider.

One major challenge was establishing a central data repository that meets the high security requirements for critical data. Which cloud provider is the right one? Which provider meets the criteria in terms of privacy and security and offers the infrastructure to properly implement the desired architecture? To answer these security-related questions while maintaining compliance with applicable regulations, the tech company sought a qualified IT security partner. Based on recommendations from its own network, the company became aware of Cloudyrion and reached out to the IT security specialists. The key advantages of the Düsseldorf-based security experts: comprehensive consulting, technological expertise, and solution-oriented, pragmatic assistance from a single source.

Challenge

Although the company had already selected a Big Data environment, it had not yet undergone a security assessment. Therefore, a systematic reverse engineering approach was needed to ensure that the three key security-relevant areas in Big Data analytics were effectively addressed and covered:

• Minimizing attack surfaces through configuration and vulnerability analysis
• Secure data transmission
• Secure data storage

Data encryption

The focus was on access permissions, data encryption during transmission, and data integration patterns that standardize which data can flow in and out and who has access to the data lakes (cold/warm storage) and BI tools. A practical framework for implementing secure role management was to be based on the „need-to-know“ principle, meaning that each individual working with the data only has visibility and access to what they need for their specific task. In Big Data environments, managing access permissions often tends to become confusing, which can be a dangerous situation.

Beyond the legal requirements, such as those embedded in the GDPR, data security is fundamentally a high priority. Once violations, negligence, or security incidents become public, the brand’s reputation is severely damaged and difficult to repair. Furthermore, compromising particularly sensitive data sets can significantly harm the company’s core business, potentially even threatening its survival. With a clear focus on security, supported by Cloudyrion, role management was to bring clarity regarding data security and compliance. The project’s timeline was set for six months, during which the high availability of data for authorized users was to be ensured at all times.

Solution

At the start of the project, Cloudyrion worked with the company to define the data integration patterns for the Big Data environment. The focus was on the logic behind the dashboards in use and who was authorized to modify them when necessary. The security experts provided guidance on the following topics, among others:

• Establishing data transparency through a data asset inventory/data catalog (missing data discovery and mapping)
• Data classification on four levels (C1 to C4)
• Data protection (data leakage/loss prevention)
• Secure data source integration and validation
• Securing data lakes (ETL and ML pipelines)
• Masking or anonymizing data sets
• Creating an asset inventory in the form of a data catalog
• Data retention management
• Secure data backup
• Identity access management

The central question for identity management was: Which roles exist? How are roles assigned, revoked, and released? How can the entire process be audited? It was important to ensure that all processes reviewed and, if necessary, adjusted according to the aforementioned methodology were compliance-relevant for the company.

In addition, the experts identified vulnerabilities and issues in the already implemented live environment, correcting them iteratively. Audits and penetration tests were conducted to find attack surfaces and close gaps. This process involved systematically reverse engineering through all layers, back to the original design of the solution, which was then adjusted based on the findings according to the security-by-design approach. To ensure the high availability of data and data lakes, the Big Data solution was originally deployed in a multi-cloud environment. Most common cloud service providers support this structurally, but their security tools tend to have a relatively high rate of false positives when identifying valid data patterns—an issue the team learned to handle with Cloudyrion’s support.

Future Outlook

After reaching the milestone set for the six-month period, the company extended Cloudyrion’s engagement to continue driving and monitoring the implementation of the security-by-design principles beyond the initial project timeframe. This deepened the trust between the client teams and the consultants. As a result, the company’s overall awareness of security issues increased significantly. This is also due to unannounced audits that Cloudyrion regularly conducts within the expanded system environment to continually identify and close vulnerabilities. In parallel, Cloudyrion remains available for support with particularly complex issues.

Benefits and Assessment

In practical terms, one challenge was teaching users the necessary skill set to work confidently with the system. However, the learning curve showed that this effort was worthwhile. „Security-related mistakes made by employees are typically not intentional but rather unconscious. The goal, however, must be to ensure that such errors never reach the production environment,“ explains Okay Güler, Founder & IT Security Consultant at Cloudyrion. „Our client and its staff particularly appreciate our solution-oriented approach and flexibility, which helps them develop and internalize security know-how in the Big Data context and sustainably anchor it in their processes.“

Key Results Overview

• Secure data storage and access
• Adherence to internal company policies
• GDPR compliance
• Implementation of „what good looks like“ data integration patterns
• Significantly faster processes
• Validation of incoming data with a security focus
• Building security know-how among staff
• Clear assignment of tasks and roles for users