RESOURCES

1. Participation in CloudNative SecurityCon 2024

 

To enhance Megazone Cloud’s global presence and acquire expertise in cloud security, Danny and I prepared to present at CloudNative SecurityCon 2024, a security conference hosted by the Linux Foundation from February.

 

Researching for the conference proved challenging due to limited available information beyond past sessions, which made it challenging to determine the most suitable content for the presentation. Additionally, our limited experience with cloud-native open-source technologies, required extra time for information verification. Balancing these tasks with our regular work made a tight preparation schedule.

 

By late march, We registered for a presentation session at the topic “Observability + Detections + Response.”, focusing on “Monitoring and Responding to Cloud-Native Vulnerabilities, Attacks, and Bugs, Minimizing Downtime, Discovering the Root Cause, and Preventing Recurrences.”

 

 

Our proposal outlined a cost-effective monitoring system using eBPF via Tetragon for kernel-level events and Fluentbit for collecting and sending data. This architecture aimed to establish efficient dashboards using OpenSearch and Grafana for detailed and diverse analysis.

 

After submitting our application, speakers’ details, and the presentation topic, we focused on preparing our presentation materials meeting the submission deadline set one week before SecurityCon.

 

 

Unfortunately, by mid-May, we learned that our presentation proposal was not selected for SecurityCon 2024, thus preventing us from presenting at the event this year.

 

 

Thankfully, we had the opportunity to attend SecurityCon to gain experience and acquire relevant technical skills. Participating in this conference allows us to strengthen our presentation content, enhance our offerings to clients, and gain valuable experience with the goal of presenting as speakers in the future.

 

 

• Attendance at SecurityCon 2024: Insights and Highlights

We attended CNCF SecurityCon 2024 from June 26th to 27th, 2024, held at the Seattle Convention Center SUMMIT. This center, completed in 2023, is expected to host various events due to its location in Seattle, where AWS headquarters is located.

 

 

SecurityCon 2024 featured keynotes and sessions spread across six medium-sized classrooms and two large halls. Ahead of the event, we selected sessions of interest and typically attended around six sessions per day, aside from the keynote.

 

 

Each morning, we enrolled in 6 keynotes and 2 sessions, followed by 4 sessions after lunch. During breaks, we visited sponsor booths where we engaged in discussions and asked questions on topics of interest.

 

 

Overall, the venue size was smaller than expected, approximately one-third to one-fourth the scale of AWS SEOUL SUMMIT 2024, and the atmosphere was quieter compared to other CNCF events.

 

 

Notable participants at the booths included Sysdig, ARMO, Isovalent, OpenSearch, Teleport, and several others.

 

 

 

2. Contents of the Sessions

 

The sessions did not delve deeply into technical content. However, we gained an understanding of the conference atmosphere, as well as the ideal and culture pursued by CNCF. This insight will enable us to make better proposals when we apply to speak as presenters next time. We would like to introduce a few sessions that left a strong impression on us.

 

A. Technical Session

 

[CSI Forensics : Unraveling Kubernetes Crime Scenes]

 

• As a security engineer with limited experience in forensics, I found this session very interesting. I expected it to cover how forensics is conducted when actual problems occur, but the session focused on automating information collection for forensics.

 

 

The Kubernetes CheckPoint API was first released in late 2022. It allows for creating stateful replicas of running containers, which can be used to back up information needed for forensics and to reduce runtime for containers that initially take a long time to start.

 

The core of this session is the capability of the Kubernetes CheckPoint API to be invoked to store replicas when forensic action is needed.

 

This invocation is conditioned by rules in Falco, a cloud-native security tool designed for Linux systems. It provides real-time alerts using custom rules enhanced with container and Kubernetes metadata on kernel events. Falco contributes to comprehensive runtime security by ensuring visibility into anomalous behavior, potential security threats, and compliance violations. Many products adopt Falco as an engine for CNAPP, with Sysdig being a prominent example.

 

For example, you can configure rules in Falco that detect communication with IPs known to be vulnerable from lists. When communication matching these rules is detected, you can use Argo workflows to invoke the Kubernetes CheckPoint API.

The CheckPoint API creates files in TAR format. These compressed files can be registered in a registry using Buildah and later restored.

 

Restored containers can be analyzed using various analysis tools, particularly leveraging the open-source tool checkpointctl. Thus, in addition to traditional log auditing, actively utilizing checkpoint copies for static and dynamic diagnostics enables proactive responses to security incidents.

 

If a large amount of data accumulates from the analysis, automating this aspect would allow for efficient operation of Kubernetes security management.

 

A table below summarizes tools used for automation and their descriptions:

 

 

[Proactive Kubernetes Security : Anomaly Detection and Runtime Alerting in Kubernetes Workloads]

 

ARMO, an open-source security platform, presented a similar approach to our eBPF-based monitoring using Tetragon and Fluentbit. They highlighted Kubescape, which is based on the PoC(Proof of Concept) called KubeCop, demonstrating real-time threat detection and response in Kubernetes environments. KubeCop establishes a baseline profile of applications and detects anomalies based on this profile (Anomaly Detection). Its key features include signature-based and anomaly-based detection of malicious behavior, as well as host malware scanning using ClamAV, an open-source antivirus engine.

 

 

This project can send alerts to various external endpoints and also send metrics to Prometheus. By applying rules, it integrates with systems like AlertManager to enable rapid response to threats.

For those interested in reviewing KubeCop’s details and installing it, you can check out the content via the GitHub link below or install it using Helm:

 

Reference : https://github.com/armosec/kubecop?tab=readme-ov-file

This provides a way to explore KubeCop’s functionalities and proceed with installation if desired.

 

[Leveraging CNCF Projects Against Kubernetes’ OWASP Top 10]

 

OWASP is an open-source web application security project that regularly publishes 10 major application vulnerabilities, which many engineers and developers  are likely familiar with. In the session, how to address these 10 vulnerabilities in Cloud Native Projects is discussed.

 

K01: Insecure Workload Configurations

K02: Supply Chain Vulnerabilities

K03: Overly Permissive RBAC Configurations

K04: Lack of Centralized Policy Enforcement

K05: Inadequate Logging and Monitoring

K06: Broken Authentication Mechanisms

K07: Missing Network Segmentation Controls

K08: Secrets Management Failures

K09: Misconfigured Cluster Components

K10: Outdated and Vulnerable Kubernetes Components

 

K01: Insecure Workload Configurations

• Insecure configurations can lead to vulnerabilities. According to 2022 data, approximately 53% reported experiencing these vulnerabilities.
• Significant impacts of misconfigurations include:
  • Running the application process as Root
  • Not considering read-only file systems based on the purpose
  • Setting indiscriminate privileges
  • Failing to consider resource constraints
  • Relevant CNCF Project
    • Kyverno
    • Kubescape
    • Open Policy Agent
    • Falco
    • KUBEWARDEN

 

K02: Supply Chain Vulnerabilities

• One of the biggest changes in transitioning from legacy systems to cloud systems is diversification.
• During the process of building and deploying images, numerous packages are installed, some of which may not guarantee reliability.
• Even highly trusted products may be compromised, leading to image tampering through backdoors. Therefore, comprehensive reviews of images, software, and packages are essential.
• Relevant CNCF Project
  • Kyverno
  • Open Policy Agent
  • Falco
  • HARBOR
  • TUF
  • in-toto

 

K03: Overly Permissive RBAC Configurations

• Role-Based Access Control (RBAC) is a recommended standard for authorization.
• It’s crucial to grant only necessary permissions and continuously review and adjust them.
• Managing RBAC in Kubernetes follows the same principles. Particularly, unnecessary use of cluster-admin privileges can significantly impact the entire infrastructure in the event of a security breach.
• Relevant CNCF Project
  • Kyverno
  • Open Policy Agent
  • Falco
  • KEYCLOAK

 

K04: Lack of Centralized Policy Enforcement

• Continuous monitoring, review, and improvement are necessary for infrastructure safety.
• Without these practices, detecting and responding to problems promptly becomes difficult.
• It’s also important to block requests for creation not only for incorrectly configured workloads but also at the object level.
• Relevant CNCF Project
  • Kyverno
  • Open Policy Agent
  • Falco
  • argo
  • flux

 

K05: Inadequate Logging and Monitoring

• Logging in Kubernetes is essential.
• When issues arise, it’s necessary to review whether all logs needed to identify the cause have been collected.
• To prevent tampering or deletion of logs, they must be stored separately, and regular analysis should be conducted.
• Recommended logs to enable:
  • Kubernetes Events
  • Application & Container Logs
  • Operating System Logs
  • Cloud Provider Logs
  • Network Logs
• Relevant CNCF Project
  • OpenTelemetry
  • cortex
  • flentd
  • JAEGER
  • Prometheus

 

K06: Broken Authentication Mechanisms

• The importance of authentication cannot be overstated. If the authentication process is not secure, protecting our assets would become challenging.
• In Kubernetes, authentication entities can be divided into Users and Service Accounts. Recommended guidelines include:
  • Avoid using certificates for end-user authentication.
  • Do not create self-signed certificates.
  • Always use Multi-Factor Authentication (MFA).
  • Do not use service account tokens from outside the cluster.
  • Use short-lived tokens for user and external service authentication.
• Relevant CNCF Project
  • KEYCLOAK
  • Falco

 

K07: Missing Network Segmentation Controls

• Following traditional network security methodologies, we segregate network areas of our infrastructure. Critical data infrastructure resides in a Private zone with restricted external communication, while service infrastructures that require external communication are in a Public zone.
• This segregation aims to prevent intrusion into critical data from external sources.
• The same approach applies when using Kubernetes. Depending on the data flow, we need to distinguish between sections where communication should occur and where it should not, and establish network policies accordingly.
• Relevant CNCF Project
  • Istio
  • cilium
  • LINKERD

 

K08: Secrets Management Failures

• In Kubernetes, Secrets are small objects containing sensitive data such as passwords or tokens.
• It’s important to consider encryption for both the values and the storage disk. Using RBAC to minimize access permissions and implementing logging and auditing of access to Secrets are also crucial.
• Logging and auditing access to Secrets should also be implemented.
• Relevant CNCF Project
  • LINKERD
  • SPIRE
  • spiffe

 

K09: Misconfigured Cluster Components

• In a Kubernetes environment where infrastructure is maintained using configuration values, the importance of these settings is extremely high.
• Continuous monitoring of configuration values is essential to ensure that the infrastructure is created and maintained as intended. It’s crucial to manage and oversee these settings regularly to maintain the desired state of the infrastructure.
• Relevant CNCF Project
  • Kyverno
  • Open Policy Agent
  • Falco
  • KEYCLOAK
  • spiffe

 

K10: Outdated and Vulnerable Kubernetes Components

• Even in legacy systems, patch management is crucial for eliminating known vulnerabilities, although it can be challenging to implement in operational environments.
• Through various methodologies, we must strive to remove the most vulnerable components possible.
• Utilizing CVE databases and scanning tools is essential for continuously managing vulnerabilities.
• Relevant CNCF Project
  • flux
  • argo
  • Falco
  • Kubescape

 

For detailed information about OWASP Top 10, please refer to the link below.
Reference : https://owasp.org/www-project-top-ten/

 

 

B. Non-technical session

 

[Championing Security: Generic Presentation Scaling Security At Every Level]

 

• Have you heard of a Security Champion? A Security Champion is designated within non-security departments to enhance security awareness and communication, contributing to the company’s security level through regular educational opportunities and rewards.

 

• Many recent security breaches are often caused by employee errors. In the session, examples were cited where phishing URLs were shared on Slack or Jira.

 

• Improving employees’ security awareness is a crucial focus in Korean compliance discussions, which I found particularly interesting. How can we effectively maintain such a Security Champion program?

 

• Here are the key requirements for maintaining a successful Security Champion program:

 

a . Identify Target Teams

• Assess the workflows and communication processes of target teams, understand their current security status and technologies used, and collect data to diagnose their current security level and set goals.

 

b. Define Roles

• Define the roles that Security Champions must fulfill based on the findings from the identification process or the activities required to reach the target level.
• Typical activities include providing feedback on security standards and procedures, enhancing team members’ security awareness, performing security code reviews, and using security testing tools.

 

c. Recommend Champions

• Management approval is required for the roles, responsibilities, and time commitment (20% of the role) of Security Champions.
• Champions can come from roles such as developers, operations, or QA.
• Each security officer can support up to five Security Champions, who are encouraged to support their teams. It is recommended to leverage specific technical skills for each platform.
• When appointing Security Champions, discuss their roles and benefits to determine if they are suitable candidates. Once designated, inform the organization about the Security Champion program and its members.

 

d. Establish Communication Channels

• Create channels where security officers and Security Champions can freely discuss and adjust goals as needed through regular meetings to discuss all issues.

 

e. Build a Robust Knowledge Base

• A single database for referencing security guidelines is essential. Share all knowledge materials created from organizational security policies, procedures, and activities to enable all members to refer to them for their work.

 

f. Sustain Interest

• Motivate Security Champions to promote security activities with ongoing passion and support continuous sharing of achievements within the company.
• Officially assign responsibilities and reflect achievements to motivate individuals to participate as Security Champions.
• Also, set goals alongside Security Champions’ security goals and key outcomes to continually monitor progress.

 

Many security engineers and managers in Korea spend considerable time and effort on compliance maintenance, making it difficult to actively pursue improvement activities. However, given that many security incidents occur due to internal errors, improving employees’ security levels through such security activities could effectively enhance practical corporate security levels.

 

 

 

3. Reflection on SecurityCon 2024

 

SecurityCon, a part of CNCF events, focuses on security within the realm of CloudNative technologies. Although we haven’t attended many conferences, we found the topics presented by speakers with diverse security experiences to be intriguing during the brief yet impactful two-day conference.

Before applying to present, we wondered how beneficial it would have been to have participated as contributors to various open-source projects. The Summit provided an opportunity to expand our network, connecting with solution providers who are difficult to meet face-to-face in Korea, as well as representatives from diverse overseas companies.

Compared to Korea, we were impressed by the more relaxed presentation style and the engaging atmosphere, which included humor such as entertaining GIFs and background music from phones. There were moments when we realized we had been too narrow in our thinking about topics, prompting us to consider preparing topics that could more broadly showcase our expertise.

We learned about many CNCF open-source projects and had the opportunity to meet founders and CNCF members, which further motivated us for future developments.