HHF's Posts

Real-World Applications of Firecracker 🔥

Wed, 08 Jan 2025 00:00:00 GMT

Introduction

While understanding the technical aspects of Firecracker is crucial, seeing how it can be applied in real-world scenarios helps demonstrate its true potential. This guide explores practical implementations and use cases that showcase Firecracker's capabilities in different environments.

Scenario 1: Continuous Integration Environment

Let's create a CI environment where each build runs in its own isolated MicroVM. This ensures complete isolation between builds and prevents any potential interference.

#!/bin/bash

# Configuration
VM_ID=$1
BUILD_ID=$2
KERNEL="/var/lib/firecracker/kernel/vmlinux-5.10"
ROOTFS="/var/lib/firecracker/rootfs/debian-ci.ext4"

# Create network interface
ip tuntap add dev tap${VM_ID} mode tap
ip link set tap${VM_ID} up
brctl addif br0 tap${VM_ID}

# Start Firecracker
cat <<EOF > /tmp/vm${VM_ID}-config.json
{
    "boot-source": {
        "kernel_image_path": "${KERNEL}",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
    },
    "drives": [
        {
            "drive_id": "rootfs",
            "path_on_host": "${ROOTFS}",
            "is_root_device": true,
            "is_read_only": false
        }
    ],
    "network-interfaces": [
        {
            "iface_id": "eth0",
            "guest_mac": "AA:FC:00:00:${VM_ID}:01",
            "host_dev_name": "tap${VM_ID}"
        }
    ],
    "machine-config": {
        "vcpu_count": 2,
        "mem_size_mib": 2048
    }
}
EOF

# Run the build in isolated environment
firecracker --api-sock /tmp/firecracker-${VM_ID}.socket --config-file /tmp/vm${VM_ID}-config.json

This setup ensures each build runs in complete isolation, preventing issues like cached dependencies or leftover processes from affecting subsequent builds.

Scenario 2: Development Sandbox Environment

Create an automated system for developers to spin up isolated development environments quickly.

import os
import json
import subprocess
from flask import Flask, request, jsonify

app = Flask(__name__)

class SandboxManager:
    def __init__(self):
        self.base_kernel = "/var/lib/firecracker/kernel/vmlinux-5.10"
        self.base_rootfs = "/var/lib/firecracker/rootfs/dev-environment.ext4"
        self.vm_configs = {}

    def create_sandbox(self, dev_id, project_type):
        # Create a copy of base rootfs for this developer
        dev_rootfs = f"/var/lib/firecracker/rootfs/dev-{dev_id}.ext4"
        subprocess.run(["cp", self.base_rootfs, dev_rootfs])

        # Configure network
        tap_device = f"tap{dev_id}"
        subprocess.run([
            "ip", "tuntap", "add", 
            "dev", tap_device, 
            "mode", "tap"
        ])
        subprocess.run(["ip", "link", "set", tap_device, "up"])
        subprocess.run(["brctl", "addif", "br0", tap_device])

        # Create VM configuration
        config = {
            "boot-source": {
                "kernel_image_path": self.base_kernel,
                "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
            },
            "drives": [{
                "drive_id": "rootfs",
                "path_on_host": dev_rootfs,
                "is_root_device": True,
                "is_read_only": False
            }],
            "network-interfaces": [{
                "iface_id": "eth0",
                "guest_mac": f"AA:FC:00:00:{dev_id:02x}:01",
                "host_dev_name": tap_device
            }],
            "machine-config": {
                "vcpu_count": 4,
                "mem_size_mib": 4096
            }
        }

        # Store configuration
        self.vm_configs[dev_id] = config
        return config

sandbox_manager = SandboxManager()

@app.route('/sandbox', methods=['POST'])
def create_sandbox():
    data = request.get_json()
    dev_id = data.get('dev_id')
    project_type = data.get('project_type')
    
    config = sandbox_manager.create_sandbox(dev_id, project_type)
    return jsonify({"status": "success", "config": config})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Scenario 3: Security Testing Environment

Create isolated environments for security testing and vulnerability scanning.

#!/bin/bash

# Configuration
LAB_NAME=$1
TOOLS_LIST=$2

# Create a fresh rootfs for security testing
dd if=/dev/zero of=/tmp/security-lab.ext4 bs=1G count=20
mkfs.ext4 /tmp/security-lab.ext4

# Mount and prepare the rootfs
mkdir -p /mnt/security-lab
mount /tmp/security-lab.ext4 /mnt/security-lab

# Install base system and security tools
debootstrap --include=$(echo $TOOLS_LIST | tr ',' ' ') \
    bullseye /mnt/security-lab http://deb.debian.org/debian/

# Configure network isolation
cat <<EOF > /mnt/security-lab/etc/network/interfaces
auto eth0
iface eth0 inet static
    address 172.16.0.2
    netmask 255.255.255.0
    gateway 172.16.0.1
EOF

# Add security hardening
cat <<EOF > /mnt/security-lab/etc/sysctl.d/99-security.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
EOF

# Prepare VM configuration
cat <<EOF > /tmp/security-lab-config.json
{
    "boot-source": {
        "kernel_image_path": "/var/lib/firecracker/kernel/vmlinux-5.10",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off ip=172.16.0.2::172.16.0.1:255.255.255.0::eth0:off"
    },
    "drives": [
        {
            "drive_id": "rootfs",
            "path_on_host": "/tmp/security-lab.ext4",
            "is_root_device": true,
            "is_read_only": false
        }
    ],
    "network-interfaces": [
        {
            "iface_id": "eth0",
            "guest_mac": "AA:FC:00:00:00:01",
            "host_dev_name": "tap0"
        }
    ],
    "machine-config": {
        "vcpu_count": 2,
        "mem_size_mib": 4096
    }
}
EOF

Scenario 4: Serverless Function Testing

Create an environment for testing serverless functions locally with realistic isolation.

import os
import json
import uuid
import asyncio
from aiohttp import web

class FunctionRunner:
    def __init__(self):
        self.vm_pool = []
        self.max_vms = 10
        self.base_config = self._load_base_config()

    def _load_base_config(self):
        return {
            "boot-source": {
                "kernel_image_path": "/var/lib/firecracker/kernel/vmlinux-5.10",
                "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
            },
            "drives": [{
                "drive_id": "rootfs",
                "path_on_host": "/var/lib/firecracker/rootfs/lambda-base.ext4",
                "is_root_device": True,
                "is_read_only": True
            }],
            "machine-config": {
                "vcpu_count": 1,
                "mem_size_mib": 128
            }
        }

    async def create_function_vm(self, function_code):
        vm_id = str(uuid.uuid4())
        
        # Create overlay filesystem for this function
        overlay_path = f"/tmp/function-{vm_id}.ext4"
        os.system(f"dd if=/dev/zero of={overlay_path} bs=1M count=512")
        os.system(f"mkfs.ext4 {overlay_path}")

        # Configure VM
        config = self.base_config.copy()
        config["drives"].append({
            "drive_id": "overlay",
            "path_on_host": overlay_path,
            "is_root_device": False,
            "is_read_only": False
        })

        # Start VM and return handle
        return await self._start_vm(vm_id, config)

    async def _start_vm(self, vm_id, config):
        # Implementation details for starting VM
        pass

    async def execute_function(self, function_code, event):
        vm = await self.create_function_vm(function_code)
        try:
            result = await vm.invoke(event)
            return result
        finally:
            await vm.cleanup()

app = web.Application()
runner = FunctionRunner()

async def handle_function(request):
    data = await request.json()
    result = await runner.execute_function(
        data['function_code'],
        data['event']
    )
    return web.json_response(result)

app.router.add_post('/functions/invoke', handle_function)

if __name__ == '__main__':
    web.run_app(app)

Scenario 5: Multi-tenant Application Testing

Create an environment for testing multi-tenant applications with proper isolation.

import asyncio
import docker
from typing import Dict, List
import yaml

class TenantEnvironment:
    def __init__(self, tenant_id: str, config: Dict):
        self.tenant_id = tenant_id
        self.config = config
        self.networks: List[str] = []
        self.vms: List[str] = []

    async def setup(self):
        # Create isolated network for tenant
        network_config = await self._create_network()
        
        # Setup application components
        await self._setup_components()
        
        # Configure monitoring
        await self._setup_monitoring()

    async def _create_network(self):
        # Implementation for creating isolated network
        pass

    async def _setup_components(self):
        # Setup different application components in isolated VMs
        pass

    async def _setup_monitoring(self):
        # Setup monitoring for this tenant's environment
        pass

class MultiTenantTestManager:
    def __init__(self, config_file: str):
        self.config = self._load_config(config_file)
        self.tenant_environments: Dict[str, TenantEnvironment] = {}

    def _load_config(self, config_file: str) -> Dict:
        with open(config_file) as f:
            return yaml.safe_load(f)

    async def create_tenant_environment(self, tenant_id: str):
        env = TenantEnvironment(tenant_id, self.config)
        await env.setup()
        self.tenant_environments[tenant_id] = env
        return env

    async def run_tests(self, test_suite: str):
        # Run tests across all tenant environments
        pass

    async def cleanup(self):
        # Cleanup all tenant environments
        pass

Scenario 6: Edge Computing Simulation

Create an environment to simulate edge computing scenarios with multiple interconnected MicroVMs.

import asyncio
import networkx as nx
from dataclasses import dataclass
from typing import Dict, List

@dataclass
class EdgeNode:
    node_id: str
    location: tuple
    capacity: Dict
    connections: List[str]
    vm_config: Dict

class EdgeComputeSimulator:
    def __init__(self, topology_file: str):
        self.topology = self._load_topology(topology_file)
        self.nodes: Dict[str, EdgeNode] = {}
        self.network = nx.Graph()

    def _load_topology(self, topology_file: str):
        # Load network topology from file
        pass

    async def create_edge_node(self, node_config: Dict):
        # Create Firecracker VM for edge node
        node_id = node_config['id']
        
        # Configure networking for this node
        network_config = self._create_network_config(node_config)
        
        # Create VM configuration
        vm_config = {
            "boot-source": {
                "kernel_image_path": "/var/lib/firecracker/kernel/vmlinux-5.10",
                "boot_args": f"console=ttyS0 reboot=k panic=1 pci=off ip={network_config['ip']}"
            },
            "drives": [{
                "drive_id": "rootfs",
                "path_on_host": f"/var/lib/firecracker/rootfs/edge-{node_id}.ext4",
                "is_root_device": True,
                "is_read_only": False
            }],
            "network-interfaces": [{
                "iface_id": "eth0",
                "guest_mac": f"AA:FC:00:{node_id:02x}:00:01",
                "host_dev_name": f"tap{node_id}"
            }],
            "machine-config": {
                "vcpu_count": node_config.get('vcpu', 1),
                "mem_size_mib": node_config.get('memory', 512)
            }
        }
        
        # Create and store node
        node = EdgeNode(
            node_id=node_id,
            location=node_config['location'],
            capacity=node_config['capacity'],
            connections=node_config['connections'],
            vm_config=vm_config
        )
        self.nodes[node_id] = node
        return node

    def _create_network_config(self, node_config: Dict):
        # Create network configuration for edge node
        ip = f"172.16.{node_config['subnet']}.{node_config['host']}/24"
        return {
            "ip": ip,
            "gateway": f"172.16.{node_config['subnet']}.1"
        }

    async def simulate_network_conditions(self):
        # Simulate different network conditions between nodes
        for edge1, edge2 in self.network.edges():
            # Apply network latency and bandwidth constraints
            latency = self.network[edge1][edge2].get('latency', '20ms')
            bandwidth = self.network[edge1][edge2].get('bandwidth', '100mbit')
            
            os.system(f"tc qdisc add dev {edge1}-{edge2} root netem delay {latency}")
            os.system(f"tc qdisc add dev {edge1}-{edge2} root tbf rate {bandwidth} burst 32kbit latency 400ms")

    async def deploy_application(self, app_config: Dict):
        # Deploy application across edge nodes
        for node_id, deployment in app_config['deployments'].items():
            node = self.nodes[node_id]
            await self._deploy_to_node(node, deployment)

    async def _deploy_to_node(self, node: EdgeNode, deployment: Dict):
        # Implementation for deploying to specific node
        pass

# Example usage
async def main():
    simulator = EdgeComputeSimulator('topology.yaml')
    
    # Create edge nodes
    nodes = [
        {'id': '01', 'location': (40.7128, -74.0060), 'subnet': 1, 'host': 2},
        {'id': '02', 'location': (34.0522, -118.2437), 'subnet': 1, 'host': 3},
        {'id': '03', 'location': (51.5074, -0.1278), 'subnet': 1, 'host': 4}
    ]
    
    for node_config in nodes:
        await simulator.create_edge_node(node_config)
    
    # Simulate network conditions
    await simulator.simulate_network_conditions()
    
    # Deploy application
    app_config = {
        'deployments': {
            '01': {'components': ['web', 'cache']},
            '02': {'components': ['database']},
            '03': {'components': ['analytics']}
        }
    }
    await simulator.deploy_application(app_config)

if __name__ == '__main__':
    asyncio.run(main())

Scenario 7: Microservices Testing Environment

Create an environment for testing microservices interactions with proper isolation and network control.

import asyncio
import yaml
from typing import Dict, List
from dataclasses import dataclass

@dataclass
class ServiceConfig:
    name: str
    version: str
    dependencies: List[str]
    resources: Dict
    config: Dict

class MicroservicesTestEnvironment:
    def __init__(self, config_file: str):
        self.config = self._load_config(config_file)
        self.services: Dict[str, ServiceConfig] = {}
        self.vms: Dict[str, Dict] = {}

    def _load_config(self, config_file: str) -> Dict:
        with open(config_file) as f:
            return yaml.safe_load(f)

    async def create_service_vm(self, service: ServiceConfig):
        """Create a Firecracker VM for a specific microservice"""
        vm_id = f"{service.name}-{service.version}"
        
        # Create rootfs for service
        rootfs_path = f"/var/lib/firecracker/rootfs/{vm_id}.ext4"
        self._prepare_service_rootfs(service, rootfs_path)
        
        # Configure VM
        config = {
            "boot-source": {
                "kernel_image_path": "/var/lib/firecracker/kernel/vmlinux-5.10",
                "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
            },
            "drives": [{
                "drive_id": "rootfs",
                "path_on_host": rootfs_path,
                "is_root_device": True,
                "is_read_only": False
            }],
            "network-interfaces": [{
                "iface_id": "eth0",
                "guest_mac": f"AA:FC:00:{hash(vm_id) % 256:02x}:00:01",
                "host_dev_name": f"tap-{vm_id}"
            }],
            "machine-config": {
                "vcpu_count": service.resources.get('cpu', 1),
                "mem_size_mib": service.resources.get('memory', 512)
            }
        }
        
        self.vms[vm_id] = config
        return config

    def _prepare_service_rootfs(self, service: ServiceConfig, rootfs_path: str):
        # Prepare rootfs with service dependencies and configuration
        pass

    async def setup_service_network(self, service: ServiceConfig):
        """Configure network for service and its dependencies"""
        vm_id = f"{service.name}-{service.version}"
        
        # Create tap device for service
        os.system(f"ip tuntap add dev tap-{vm_id} mode tap")
        os.system(f"ip link set tap-{vm_id} up")
        os.system(f"brctl addif br0 tap-{vm_id}")

        # Setup network policies for service dependencies
        for dependency in service.dependencies:
            self._setup_network_policy(service.name, dependency)

    def _setup_network_policy(self, service: str, dependency: str):
        # Implement network policies between services
        pass

    async def deploy_services(self):
        """Deploy all services in the configuration"""
        for service_name, service_config in self.config['services'].items():
            service = ServiceConfig(
                name=service_name,
                version=service_config['version'],
                dependencies=service_config.get('dependencies', []),
                resources=service_config.get('resources', {}),
                config=service_config.get('config', {})
            )
            
            # Create and configure VM for service
            await self.create_service_vm(service)
            await self.setup_service_network(service)

    async def run_integration_tests(self):
        """Run integration tests across services"""
        # Implementation for running tests
        pass

# Example usage
async def main():
    env = MicroservicesTestEnvironment('services-config.yaml')
    await env.deploy_services()
    await env.run_integration_tests()

if __name__ == '__main__':
    asyncio.run(main())

Scenario 8: Load Testing Environment

Create an environment for performing load tests with proper resource isolation.

import asyncio
import statistics
from dataclasses import dataclass
from typing import Dict, List

@dataclass
class LoadTestConfig:
    target_url: str
    concurrent_users: int
    ramp_up_time: int
    test_duration: int
    scenarios: List[Dict]

class LoadTestEnvironment:
    def __init__(self, config: LoadTestConfig):
        self.config = config
        self.vms: List[str] = []
        self.results: List[Dict] = []

    async def setup_load_generators(self):
        """Create VMs for load generation"""
        vms_needed = self._calculate_required_vms()
        
        for i in range(vms_needed):
            vm_id = f"loadgen-{i:03d}"
            
            # Create VM configuration
            config = {
                "boot-source": {
                    "kernel_image_path": "/var/lib/firecracker/kernel/vmlinux-5.10",
                    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
                },
                "drives": [{
                    "drive_id": "rootfs",
                    "path_on_host": "/var/lib/firecracker/rootfs/loadgen.ext4",
                    "is_root_device": True,
                    "is_read_only": True
                }],
                "network-interfaces": [{
                    "iface_id": "eth0",
                    "guest_mac": f"AA:FC:00:00:{i:02x}:01",
                    "host_dev_name": f"tap{i}"
                }],
                "machine-config": {
                    "vcpu_count": 2,
                    "mem_size_mib": 1024
                }
            }
            
            # Create network interface
            os.system(f"ip tuntap add dev tap{i} mode tap")
            os.system(f"ip link set tap{i} up")
            os.system(f"brctl addif br0 tap{i}")
            
            self.vms.append(vm_id)

    def _calculate_required_vms(self) -> int:
        """Calculate number of VMs needed based on load parameters"""
        users_per_vm = 1000  # Adjust based on your needs
        return max(1, self.config.concurrent_users // users_per_vm)

    async def run_load_test(self):
        """Execute load test across all VMs"""
        # Distribute load across VMs
        users_per_vm = self.config.concurrent_users // len(self.vms)
        
        tasks = []
        for vm_id in self.vms:
            task = asyncio.create_task(
                self._run_load_test_on_vm(vm_id, users_per_vm)
            )
            tasks.append(task)
        
        results = await asyncio.gather(*tasks)
        self.results.extend(results)

    async def _run_load_test_on_vm(self, vm_id: str, user_count: int):
        """Run load test on specific VM"""
        # Implementation for running load test on VM
        pass

    def analyze_results(self) -> Dict:
        """Analyze load test results"""
        response_times = [r['response_time'] for r in self.results]
        
        return {
            'avg_response_time': statistics.mean(response_times),
            'p95_response_time': statistics.quantiles(response_times, n=20)[18],
            'p99_response_time': statistics.quantiles(response_times, n=100)[98],
            'total_requests': len(response_times),
            'success_rate': sum(1 for r in self.results if r['success']) / len(self.results)
        }

# Example usage
async def main():
    config = LoadTestConfig(
        target_url="http://target-service:8080",
        concurrent_users=5000,
        ramp_up_time=300,
        test_duration=3600,
        scenarios=[
            {'name': 'browse_catalog', 'weight': 0.4},
            {'name': 'add_to_cart', 'weight': 0.3},
            {'name': 'checkout', 'weight': 0.3}
        ]
    )
    
    env = LoadTestEnvironment(config)
    await env.setup_load_generators()
    await env.run_load_test()
    results = env.analyze_results()
    print(f"Load Test Results: {results}")

if __name__ == '__main__':
    asyncio.run(main())

These scenarios demonstrate the versatility of Firecracker MicroVMs in different use cases. Each scenario includes detailed implementation examples and considerations for:

Resource isolation
Network configuration
Performance monitoring
Data collection and analysis
Automation and orchestration

The examples can be extended and modified based on specific requirements while maintaining the core benefits of Firecracker: rapid startup times, strong isolation, and efficient resource utilization.

Remember to always follow best practices for security and resource management when implementing these scenarios in production environments.

CLP Modern Framework Deploy & CI/CD Integration

Sun, 08 Dec 2024 00:00:00 GMT

Advanced Deployment Guide: Modern Framework Deployment & CI/CD Integration

Part 1: Advanced Framework Deployments

Astro Deployment

Let's deploy an Astro application with server-side rendering (SSR) and dynamic routes.

First, create a Node.js site for Astro:

clpctl site:add:nodejs \
    --domainName=astro.yourdomain.com \
    --nodejsVersion=18 \
    --appPort=4321 \
    --siteUser=astro-admin \
    --siteUserPassword='secure_password_here'

Set up the Astro project with SSR:

cd htdocs/astro.yourdomain.com/
# Create new Astro project
npm create astro@latest

# Add SSR adapter
npm install @astrojs/node

Configure Astro for SSR (astro.config.mjs):

import { defineConfig } from 'astro/config';
import node from '@astrojs/node';

export default defineConfig({
  output: 'server',
  adapter: node({
    mode: 'standalone'
  }),
  server: {
    port: 4321,
    host: true
  }
});

Create PM2 configuration for Astro:

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'astro-app',
    script: './dist/server/entry.mjs',
    env: {
      HOST: '127.0.0.1',
      PORT: 4321,
      NODE_ENV: 'production'
    },
    exec_mode: 'cluster',
    instances: 'max',
    max_memory_restart: '500M'
  }]
}

Next.js Advanced Deployment

Let's deploy a Next.js application with custom server configuration and internationalization.

Set up the site:

clpctl site:add:nodejs \
    --domainName=next.yourdomain.com \
    --nodejsVersion=18 \
    --appPort=3000 \
    --siteUser=next-admin \
    --siteUserPassword='secure_password_here'

Configure Next.js with custom server:

// server.js
const { createServer } = require('http');
const { parse } = require('url');
const next = require('next');

const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

app.prepare().then(() => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true);
    handle(req, res, parsedUrl);
  }).listen(3000, (err) => {
    if (err) throw err;
    console.log('> Ready on http://localhost:3000');
  });
});

Advanced PM2 configuration for Next.js:

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'nextjs-advanced',
    script: 'server.js',
    instances: 'max',
    exec_mode: 'cluster',
    watch: false,
    max_memory_restart: '1G',
    env: {
      NODE_ENV: 'production',
      PORT: 3000
    },
    env_production: {
      NODE_ENV: 'production',
      PORT: 3000
    }
  }]
}

Docusaurus Deployment

Let's deploy a Docusaurus site with versioned documentation and search functionality.

Create a static site for Docusaurus:

clpctl site:add:static \
    --domainName=docs.yourdomain.com \
    --siteUser=docs-admin \
    --siteUserPassword='secure_password_here'

Set up Docusaurus with versioning:

cd htdocs/docs.yourdomain.com/
npm init docusaurus@latest
npm run build

Configure versioning:

// docusaurus.config.js
module.exports = {
  title: 'Your Documentation',
  tagline: 'Documentation made easy',
  url: 'https://docs.yourdomain.com',
  baseUrl: '/',
  onBrokenLinks: 'throw',
  onBrokenMarkdownLinks: 'warn',
  favicon: 'img/favicon.ico',
  organizationName: 'your-org',
  projectName: 'your-docs',
  
  presets: [
    [
      '@docusaurus/preset-classic',
      {
        docs: {
          sidebarPath: require.resolve('./sidebars.js'),
          editUrl: 'https://github.com/your-org/your-docs/edit/main/',
          showLastUpdateTime: true,
          showLastUpdateAuthor: true,
        },
      },
    ],
  ],
};

Part 2: Advanced PM2 Management

Load Balancing with PM2

Cluster Mode Configuration:

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'load-balanced-app',
    script: 'server.js',
    instances: -1, // Use all available CPUs
    exec_mode: 'cluster',
    watch: true,
    max_memory_restart: '1G',
    env: {
      NODE_ENV: 'production',
      PORT: 3000
    },
    node_args: '--max_old_space_size=4096'
  }]
}

Advanced PM2 Monitoring:

# Monitor memory usage
pm2 monit

# Set up PM2 monitoring dashboard
pm2 plus

# Advanced logs with timestamp
pm2 logs --timestamp --lines 1000

# Rotate logs
pm2 install pm2-logrotate
pm2 set pm2-logrotate:max_size 10M
pm2 set pm2-logrotate:retain 7

Zero-Downtime Deployment:

# Reload without downtime
pm2 reload ecosystem.config.js --env production

# Rolling reload
pm2 reload ecosystem.config.js --env production --parallel 2

# Graceful reload
pm2 gracefulReload all

Memory Management:

// ecosystem.config.js with memory management
module.exports = {
  apps: [{
    name: 'memory-managed-app',
    script: 'server.js',
    instances: 4,
    exec_mode: 'cluster',
    max_memory_restart: '1G',
    node_args: '--max_old_space_size=4096',
    increment_var: 'PORT',
    env: {
      NODE_ENV: 'production',
      PORT: 3000
    },
    error_file: 'logs/err.log',
    out_file: 'logs/out.log',
    merge_logs: true,
    exp_backoff_restart_delay: 100
  }]
}

Part 3: GitHub CI/CD Integration

GitHub Actions Workflow for Node.js Application

Basic Deployment Workflow:

# .github/workflows/deploy.yml
name: Deploy to Production

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v2
    
    - name: Setup Node.js
      uses: actions/setup-node@v2
      with:
        node-version: '18'
        
    - name: Install Dependencies
      run: npm ci
        
    - name: Build
      run: npm run build
        
    - name: Deploy to Server
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.SERVER_HOST }}
        username: ${{ secrets.SERVER_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd ~/htdocs/your-domain.com
          git pull origin main
          npm ci
          npm run build
          pm2 reload ecosystem.config.js --env production

Advanced Workflow with Testing and Staging:

name: CI/CD Pipeline

on:
  push:
    branches: [ main, staging ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    
    - name: Setup Node.js
      uses: actions/setup-node@v2
      with:
        node-version: '18'
        
    - name: Install Dependencies
      run: npm ci
        
    - name: Run Tests
      run: npm test
        
    - name: Run Linting
      run: npm run lint
      
  deploy-staging:
    needs: test
    if: github.ref == 'refs/heads/staging'
    runs-on: ubuntu-latest
    
    steps:
    - name: Deploy to Staging
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.STAGING_HOST }}
        username: ${{ secrets.STAGING_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd ~/htdocs/staging.your-domain.com
          git pull origin staging
          npm ci
          npm run build
          pm2 reload ecosystem.config.js --env staging
          
  deploy-production:
    needs: [test]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    
    steps:
    - name: Deploy to Production
      uses: appleboy/ssh-action@master
      with:
        host: ${{ secrets.PROD_HOST }}
        username: ${{ secrets.PROD_USER }}
        key: ${{ secrets.SSH_PRIVATE_KEY }}
        script: |
          cd ~/htdocs/your-domain.com
          git pull origin main
          npm ci
          npm run build
          pm2 reload ecosystem.config.js --env production

Automated Backup Before Deployment:

# Add this step before deployment
- name: Backup Database
  uses: appleboy/ssh-action@master
  with:
    host: ${{ secrets.SERVER_HOST }}
    username: ${{ secrets.SERVER_USER }}
    key: ${{ secrets.SSH_PRIVATE_KEY }}
    script: |
      clpctl db:backup --retentionPeriod=7
      clpctl remote-backup:create

Security Best Practices

Environment Variables:

# On your CloudPanel server
nano ~/.env.production

# Add your environment variables
DATABASE_URL=postgresql://user:pass@localhost:5432/db
REDIS_URL=redis://localhost:6379
JWT_SECRET=your-secret-key

# Update PM2 config to use env file
pm2 ecosystem.config.js --env production --env-path ~/.env.production

SSL Certificate Automation:

# Add to GitHub Actions workflow
- name: Update SSL Certificate
  uses: appleboy/ssh-action@master
  with:
    host: ${{ secrets.SERVER_HOST }}
    username: ${{ secrets.SERVER_USER }}
    key: ${{ secrets.SSH_PRIVATE_KEY }}
    script: |
      clpctl lets-encrypt:renew:certificates

Monitoring Setup:

// ecosystem.config.js with monitoring
module.exports = {
  apps: [{
    name: 'monitored-app',
    script: 'server.js',
    instances: 'max',
    exec_mode: 'cluster',
    max_memory_restart: '1G',
    merge_logs: true,
    log_date_format: 'YYYY-MM-DD HH:mm:ss Z',
    error_file: 'logs/err.log',
    out_file: 'logs/out.log',
    log_type: 'json',
    watch: ['server', 'client'],
    ignore_watch: ['node_modules', 'logs']
  }]
}

Remember to always test your CI/CD pipeline in a staging environment first and maintain proper backup strategies before deploying to production. Regular monitoring and logging are crucial for maintaining a healthy production environment.

Optimizing Node.js Code in CloudPanel

Sat, 09 Nov 2024 00:00:00 GMT

I Reduced Server Costs by 52% Just by Optimizing Node.js Code in CloudPanel

The Art of Server Cost-Cutting: A Node.js Optimization Journey Part 1

It started as a shock - I opened my CloudPanel dashboard after a deep breath and saw the memory usage climbing steadily over time. I expected all Node.js applications to have some memory footprint, but this was different. Our modest application was consuming resources like it was hosting a Silicon Valley startup's entire infrastructure.

If you're just starting this journey, here's what the typical Node.js application running on CloudPanel looks like - steadily increasing RAM usage, creeping up resource utilization, and leaving your wallet feeling lighter each month as you keep upgrading server specs instead of fixing the root cause.

The Memory Leaks Expedition

The first critical area I tackled was memory leaks. When you're running Node.js applications on CloudPanel, memory leaks can be particularly sneaky because they don't immediately manifest as problems. Here's what an unoptimized memory usage pattern typically looks like:

// Before Optimization - Memory Leak Example
const cache = new Map();

async function processData(key, data) {
    // Memory leak: Continuously growing cache without cleanup
    cache.set(key, {
        data: data,
        timestamp: Date.now(),
        processedResult: await heavyComputation(data)
    });
}

setInterval(() => {
    // Periodic operation without cleanup
    processNewData();
}, 5000);

The fix involved implementing proper memory management and monitoring:

// After Optimization - Memory Management
const cache = new Map();
const MAX_CACHE_SIZE = 1000;
const CACHE_TTL = 3600000; // 1 hour in milliseconds

async function processData(key, data) {
    // Implement cache size limits
    if (cache.size >= MAX_CACHE_SIZE) {
        const oldestKey = [...cache.keys()][0];
        cache.delete(oldestKey);
    }
    
    // Add TTL to cache entries
    cache.set(key, {
        data: data,
        timestamp: Date.now(),
        processedResult: await heavyComputation(data)
    });
}

// Implement cache cleanup
setInterval(() => {
    const now = Date.now();
    for (const [key, value] of cache.entries()) {
        if (now - value.timestamp > CACHE_TTL) {
            cache.delete(key);
        }
    }
}, 300000); // Clean every 5 minutes

Event Loop Blockage Resolution

CloudPanel's performance metrics helped identify another crucial issue - event loop blocking. Here's what we found and fixed:

Before Optimization:

function processLargeDataset(data) {
    // Blocking operation
    return data.map(item => heavyTransformation(item));
}

After Optimization:

async function processLargeDataset(data) {
    const batchSize = 100;
    const results = [];
    
    for (let i = 0; i < data.length; i += batchSize) {
        const batch = data.slice(i, i + batchSize);
        // Process in batches and yield to event loop
        const batchResults = await Promise.all(
            batch.map(async item => {
                const result = await heavyTransformation(item);
                return result;
            })
        );
        results.push(...batchResults);
        // Allow other operations to process
        await new Promise(resolve => setImmediate(resolve));
    }
    return results;
}

The Results Speak Volumes

Our optimization efforts led to dramatic improvements:

Memory Usage: Dropped from 1.8GB to 256MB
Response Time: From 600ms to 250ms average
CPU Utilization: Decreased from 80% to 35%
Instance Size: Downgradable to smaller specifications
Monthly Costs: Reduced by 80%

CloudPanel-Specific Optimization Checklist

Performance Tuning

Memory Management:

// Implement garbage collection hints
if (global.gc) {
    setInterval(() => {
        global.gc();
    }, 30000);
}

Database Connection Pooling:

const pool = mysql.createPool({
    host: 'localhost',
    user: 'your_user',
    password: 'your_password',
    database: 'your_db',
    connectionLimit: 10,
    queueLimit: 0
});

CloudPanel Monitoring Setup

System Monitoring:

const monitoring = require('node-monitoring');

monitoring.watch({
    memory: true,
    cpu: true,
    eventLoop: true,
    interval: 5000 // 5 seconds
});

Custom Metrics Collection:

const metrics = {
    requestCount: 0,
    errorCount: 0,
    responseTime: []
};

app.use((req, res, next) => {
    const start = Date.now();
    metrics.requestCount++;
    
    res.on('finish', () => {
        metrics.responseTime.push(Date.now() - start);
    });
    
    next();
});

Resource Pooling

Implement connection pooling for all external resources:

const genericPool = require('generic-pool');

const factory = {
    create: async () => {
        return await createExpensiveResource();
    },
    destroy: async (client) => {
        await client.disconnect();
    }
};

const pool = genericPool.createPool(factory, {
    max: 10,
    min: 2,
    idleTimeoutMillis: 30000,
    acquireTimeoutMillis: 30000
});

Monitoring Setup in CloudPanel

Enable CloudPanel's built-in monitoring:
- CPU usage tracking
- Memory utilization metrics
- Disk I/O monitoring
- Network performance tracking

Configure custom monitoring dashboards:

const dashboard = new Dashboard({
    title: 'Node.js Application Metrics',
    refresh: '10s',
    panels: [
        memoryUsagePanel,
        cpuUsagePanel,
        responseTimePanel,
        errorRatePanel
    ]
});

Set up alerting:

const alerts = new AlertManager({
    threshold: {
        memory: 80, // Percentage
        cpu: 70,    // Percentage
        responseTime: 1000 // milliseconds
    },
    notification: {
        email: 'admin@yourdomain.com',
        slack: 'webhook_url'
    }
});

Key Resources

CloudPanel Documentation
Node.js Performance Guide
V8 Engine Optimization Documentation
Memory Leak Detection Tools

Remember: Optimization is a journey, not a destination. Start with monitoring, identify what matters most to your application, and optimize accordingly. Your stack will thank you.

Additional Tips

Use Node.js --inspect flag during development for debugging memory issues
Implement circuit breakers for external service calls
Use compression middleware for response payload optimization
Implement proper error handling and logging
Regular performance audits using CloudPanel's built-in tools

This optimization journey transformed our application from a resource-hungry monster into a lean, efficient system. The key was understanding how Node.js interacts with system resources and leveraging CloudPanel's tools for monitoring and optimization.

Optimizing PHP Code in CloudPanel is a Must

Thu, 09 Jan 2025 00:00:00 GMT

I Reduced Server spikes by 65% Just by Optimizing PHP Code in CloudPanel

The Art of Server Cost-Cutting: A PHP Optimization Journey Part 2

It began with a startling realization - I checked my CloudPanel dashboard and watched PHP-FPM's memory usage climb steadily higher for a simple php app. While PHP applications naturally consume some memory, this pattern was concerning. Our modest application was devouring resources as if it were processing an enterprise workload.

Let me walk you through my journey of optimization that not only saved costs but transformed our application's performance.

Understanding PHP Memory Management

PHP's memory management differs fundamentally from Node.js. PHP follows a shared-nothing architecture, meaning each request gets its own memory space that's cleared after the request ends. However, this doesn't make us immune to memory issues. Here's what an unoptimized PHP script typically looks like:

// Before Optimization - Memory Intensive Code
function processLargeDataset($data) {
    $results = [];
    foreach ($data as $item) {
        // Building large arrays in memory
        $results[] = heavyTransformation($item);
        // No memory limit checks
        // No garbage collection hints
    }
    return $results;
}

// Large static arrays
$cache = [];

Let's transform this into memory-efficient code:

// After Optimization - Memory Efficient Code
function processLargeDataset($data) {
    // Set memory limit for this operation
    ini_set('memory_limit', '256M');
    
    $results = [];
    $counter = 0;
    
    foreach ($data as $item) {
        $results[] = heavyTransformation($item);
        $counter++;
        
        // Periodically check memory usage
        if ($counter % 100 === 0) {
            $memoryUsage = memory_get_usage(true);
            if ($memoryUsage > 200 * 1024 * 1024) { // 200MB threshold
                // Handle memory threshold exceeded
                error_log("Memory threshold reached: " . ($memoryUsage / 1024 / 1024) . "MB");
                gc_collect_cycles(); // Force garbage collection
            }
        }
    }
    
    return $results;
}

PHP-FPM Configuration Optimization

One of the most crucial aspects of PHP performance in CloudPanel is proper PHP-FPM configuration. Here's an optimized configuration:

; php-fpm.conf optimization
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 500

; Process idle timeout
pm.process_idle_timeout = 10s

; Emergency restart threshold
emergency_restart_threshold = 10
emergency_restart_interval = 1m

; PHP memory settings
php_admin_value[memory_limit] = 256M
php_admin_value[max_execution_time] = 60
php_admin_value[max_input_time] = 60

Database Connection Management

In PHP, database connections need careful management. Here's an optimized approach:

class DatabaseConnection {
    private static $instance = null;
    private $pdo;
    
    private function __construct() {
        $this->pdo = new PDO(
            "mysql:host=localhost;dbname=your_database",
            "username",
            "password",
            [
                PDO::ATTR_PERSISTENT => true,
                PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
                PDO::ATTR_EMULATE_PREPARES => false
            ]
        );
    }
    
    public static function getInstance() {
        if (self::$instance === null) {
            self::$instance = new self();
        }
        return self::$instance;
    }
    
    public function query($sql, $params = []) {
        $stmt = $this->pdo->prepare($sql);
        $stmt->execute($params);
        return $stmt;
    }
}

Session Management Optimization

Sessions can be a significant source of server load. Here's how to optimize them:

// Custom session handler using Redis
class RedisSessionHandler implements SessionHandlerInterface {
    private $redis;
    private $ttl;
    
    public function __construct() {
        $this->redis = new Redis();
        $this->redis->connect('127.0.0.1', 6379);
        $this->ttl = 3600; // 1 hour
    }
    
    public function open($savePath, $sessionName) {
        return true;
    }
    
    public function close() {
        return true;
    }
    
    public function read($id) {
        $data = $this->redis->get("session:$id");
        return $data ?: '';
    }
    
    public function write($id, $data) {
        return $this->redis->setex("session:$id", $this->ttl, $data);
    }
    
    public function destroy($id) {
        return $this->redis->del("session:$id");
    }
    
    public function gc($maxlifetime) {
        return true; // Redis handles expiration automatically
    }
}

// Initialize custom session handler
$handler = new RedisSessionHandler();
session_set_save_handler($handler, true);

Caching Implementation

Proper caching is crucial for PHP applications:

class CacheManager {
    private $redis;
    private static $instance = null;
    
    private function __construct() {
        $this->redis = new Redis();
        $this->redis->connect('127.0.0.1', 6379);
    }
    
    public static function getInstance() {
        if (self::$instance === null) {
            self::$instance = new self();
        }
        return self::$instance;
    }
    
    public function get($key) {
        return $this->redis->get("cache:$key");
    }
    
    public function set($key, $value, $ttl = 3600) {
        return $this->redis->setex("cache:$key", $ttl, $value);
    }
    
    public function invalidate($key) {
        return $this->redis->del("cache:$key");
    }
}

Monitoring Integration

Setting up comprehensive monitoring in CloudPanel:

class ApplicationMonitor {
    private static $metrics = [];
    
    public static function recordMetric($name, $value) {
        self::$metrics[$name][] = $value;
    }
    
    public static function getAverageMetric($name) {
        if (!isset(self::$metrics[$name])) {
            return null;
        }
        return array_sum(self::$metrics[$name]) / count(self::$metrics[$name]);
    }
    
    public static function logMetrics() {
        foreach (self::$metrics as $name => $values) {
            error_log(sprintf(
                "Metric %s: Avg=%f, Min=%f, Max=%f",
                $name,
                array_sum($values) / count($values),
                min($values),
                max($values)
            ));
        }
    }
}

// Usage in application
register_shutdown_function(function() {
    ApplicationMonitor::logMetrics();
});

The Results After Optimization

Our optimization efforts yielded impressive results:

Memory Usage: Reduced from 2GB to 384MB
Response Time: Improved from 800ms to 150ms
CPU Utilization: Decreased from 75% to 30%
PHP-FPM Process Count: Optimized from 100 to 50
Monthly Costs: Reduced by 80% (Digital Ocean)

CloudPanel Configuration Best Practices

PHP Configuration:

; php.ini optimizations
opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.revalidate_freq=60
opcache.fast_shutdown=1
opcache.enable_cli=1

realpath_cache_size = 4096K
realpath_cache_ttl = 600

Nginx Configuration:

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 60s;
fastcgi_send_timeout 60s;
fastcgi_read_timeout 60s;

Monitoring Setup in CloudPanel

Enable comprehensive monitoring:

// Application-level monitoring
function monitor_application_metrics() {
    $metrics = [
        'memory_usage' => memory_get_usage(true),
        'peak_memory' => memory_get_peak_usage(true),
        'cpu_usage' => sys_getloadavg()[0],
        'php_fpm_processes' => shell_exec('ps aux | grep php-fpm | wc -l')
    ];
    
    file_put_contents(
        '/var/log/php/metrics.log',
        date('Y-m-d H:i:s') . ' ' . json_encode($metrics) . "\n",
        FILE_APPEND
    );
}

Additional Performance Tips

Implement proper error handling:

set_error_handler(function($errno, $errstr, $errfile, $errline) {
    error_log(sprintf(
        "Error [%d]: %s in %s on line %d",
        $errno,
        $errstr,
        $errfile,
        $errline
    ));
    return true;
});

Use proper autoloading:

spl_autoload_register(function ($class) {
    include __DIR__ . '/classes/' . $class . '.php';
});

Conclusion

This optimization journey transformed our PHP application from a resource-hungry system into an efficient, cost-effective solution. The key was understanding PHP's memory model, leveraging CloudPanel's tools, and implementing proper caching and connection management strategies.

Securing CloudPanel on Ubuntu 24.04 Part 1

Wed, 13 Nov 2024 00:00:00 GMT

Securing CloudPanel on Ubuntu 24.04: A Comprehensive Guide

Part 1: Initial Server Access and Core Configuration

1. Initial SSH Security Configuration

# Edit SSH configuration
sudo nano /etc/ssh/sshd_config

Add these security-focused settings:

# SSH Security Configuration
Port 2222                    # Custom SSH port
PermitRootLogin no          # Disable root login
PasswordAuthentication no   # Key-based auth only
Protocol 2                 # SSHv2 only
MaxAuthTries 3            # Limit auth attempts
ClientAliveInterval 300  # 5-min timeout
X11Forwarding no        # Disable X11 forwarding
AllowTcpForwarding no  # Disable TCP forwarding

Restart SSH:

sudo systemctl restart sshd

2. CloudPanel-Specific Firewall Rules

Configure UFW based on CloudPanel's database structure:

# Configure UFW rules based on CloudPanel's firewall_rule table
sqlite3 /home/clp/htdocs/app/data/db.sq3 << EOF
INSERT INTO firewall_rule (created_at, updated_at, port_range, source, description)
VALUES 
    (datetime('now'), datetime('now'), '8443', '0.0.0.0/0', 'CloudPanel Admin Interface'),
    (datetime('now'), datetime('now'), '80', '0.0.0.0/0', 'HTTP'),
    (datetime('now'), datetime('now'), '443', '0.0.0.0/0', 'HTTPS'),
    (datetime('now'), datetime('now'), '2222', '0.0.0.0/0', 'Custom SSH Port');
EOF

# Apply UFW rules
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 8443/tcp comment 'CloudPanel Admin'
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'
sudo ufw allow 2222/tcp comment 'SSH'

4. Database Security and Monitoring

Setup database security using CloudPanel's structure:

# Set proper permissions
sudo chown clp:clp /home/clp/htdocs/app/data/db.sq3
sudo chmod 770 /home/clp/htdocs/app/data/db.sq3

# Create automated database backup script
sudo nano /usr/local/bin/cloudpanel-db-backup.sh

#!/bin/bash

# Database backup script
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
BACKUP_DIR="${CP_HOME}/backups/database"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Create backup with proper permissions
sqlite3 "$DB_PATH" ".backup '${BACKUP_DIR}/db_${TIMESTAMP}.sq3'"
chmod 640 "${BACKUP_DIR}/db_${TIMESTAMP}.sq3"
chown clp:clp "${BACKUP_DIR}/db_${TIMESTAMP}.sq3"

# Keep only last 7 days of backups
find "$BACKUP_DIR" -name "db_*.sq3" -mtime +7 -delete

# Log backup event in CloudPanel's event table
sqlite3 "$DB_PATH" "
    INSERT INTO event (
        created_at,
        user_name,
        event_name,
        event_data
    ) VALUES (
        datetime('now'),
        'system',
        'database_backup',
        'Database backup created: db_${TIMESTAMP}.sq3'
    );"

Securing CloudPanel on Ubuntu 24.04 Part 10

Thu, 21 Nov 2024 00:00:00 GMT

Part 10: Automated Incident Response and Recovery System

Let's create a comprehensive incident response system that automatically detects and handles security incidents while maintaining proper logging in CloudPanel.

# Create incident response management script
sudo nano /usr/local/bin/cloudpanel-incident-response.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
INCIDENT_DIR="${CP_HOME}/logs/incidents"
BACKUP_DIR="${CP_HOME}/backups/incidents"

# Create necessary directories
mkdir -p "$INCIDENT_DIR" "$BACKUP_DIR"

# Function to detect and categorize security incidents
detect_security_incidents() {
    echo "Analyzing security incidents..."
    
    # Query CloudPanel's event table for potential incidents
    sqlite3 "$DB_PATH" "
        SELECT 
            e.id,
            e.created_at,
            e.user_name,
            e.event_name,
            e.event_data,
            e.source_ip_address,
            e.user_agent
        FROM event e
        WHERE e.created_at > datetime('now', '-15 minutes')
        AND e.event_name IN (
            'login_failed',
            'unauthorized_access',
            'file_modification',
            'config_change',
            'malware_detected'
        )
        ORDER BY e.created_at DESC;" | while IFS='|' read -r event_id timestamp username event_name data ip agent; do
        
        case "$event_name" in
            "login_failed")
                handle_authentication_incident "$event_id" "$ip" "$username"
                ;;
            "unauthorized_access")
                handle_access_violation "$event_id" "$ip" "$data"
                ;;
            "file_modification")
                handle_file_incident "$event_id" "$username" "$data"
                ;;
            "malware_detected")
                handle_malware_incident "$event_id" "$data"
                ;;
            *)
                log_unknown_incident "$event_id" "$event_name" "$data"
                ;;
        esac
    done
}

# Function to handle authentication-related incidents
handle_authentication_incident() {
    local event_id="$1"
    local ip="$2"
    local username="$3"
    
    # Check for brute force patterns
    local failed_attempts=$(sqlite3 "$DB_PATH" "
        SELECT COUNT(*) FROM event
        WHERE source_ip_address = '$ip'
        AND event_name = 'login_failed'
        AND created_at > datetime('now', '-1 hour');")
    
    if [ "$failed_attempts" -gt 5 ]; then
        # Implement automatic IP blocking
        sqlite3 "$DB_PATH" "
            INSERT INTO blocked_ip (
                created_at,
                updated_at,
                ip
            ) VALUES (
                datetime('now'),
                datetime('now'),
                '$ip'
            );"
        
        # Create a high-priority notification
        create_incident_notification \
            "Brute Force Attack Detected" \
            "Multiple failed login attempts from IP: $ip targeting user: $username" \
            "high"
            
        # Update firewall rules
        update_firewall_rules "$ip"
    fi
}

# Function to handle file system incidents
handle_file_incident() {
    local event_id="$1"
    local username="$2"
    local file_data="$3"
    
    # Extract file information
    local file_path=$(echo "$file_data" | jq -r '.path')
    local modification_type=$(echo "$file_data" | jq -r '.type')
    
    # Create incident snapshot
    local snapshot_dir="${BACKUP_DIR}/file_incident_$(date +%Y%m%d_%H%M%S)"
    mkdir -p "$snapshot_dir"
    
    if [ -f "$file_path" ]; then
        # Create file backup
        cp -p "$file_path" "$snapshot_dir/"
        
        # Record file metadata
        stat "$file_path" > "${snapshot_dir}/file_metadata.txt"
        
        # Check for suspicious content
        if grep -qiE "eval|base64_decode|system|exec" "$file_path"; then
            # Quarantine suspicious file
            local quarantine_dir="${CP_HOME}/quarantine/files"
            mkdir -p "$quarantine_dir"
            mv "$file_path" "${quarantine_dir}/$(basename "$file_path").$(date +%s)"
            
            create_incident_notification \
                "Suspicious File Detected" \
                "File containing potentially malicious code: $file_path" \
                "critical"
        fi
    fi
}

# Function to handle malware incidents
handle_malware_incident() {
    local event_id="$1"
    local malware_data="$2"
    
    # Extract malware information
    local infected_path=$(echo "$malware_data" | jq -r '.path')
    local malware_type=$(echo "$malware_data" | jq -r '.type')
    
    # Create quarantine directory
    local quarantine_dir="${CP_HOME}/quarantine/malware/$(date +%Y%m%d)"
    mkdir -p "$quarantine_dir"
    
    if [ -e "$infected_path" ]; then
        # Quarantine the infected file/directory
        mv "$infected_path" "$quarantine_dir/"
        
        # Record incident details
        echo "Quarantined at: $(date)" > "${quarantine_dir}/incident_details.txt"
        echo "Original location: $infected_path" >> "${quarantine_dir}/incident_details.txt"
        echo "Malware type: $malware_type" >> "${quarantine_dir}/incident_details.txt"
        
        create_incident_notification \
            "Malware Detected" \
            "Malware type '$malware_type' detected at: $infected_path" \
            "critical"
            
        # Trigger additional security scans
        initiate_security_scan "$(dirname "$infected_path")"
    fi
}

# Function to create incident notifications in CloudPanel
create_incident_notification() {
    local subject="$1"
    local message="$2"
    local severity="$3"
    
    sqlite3 "$DB_PATH" "
        INSERT INTO notification (
            created_at,
            updated_at,
            severity,
            subject,
            message,
            is_read
        ) VALUES (
            datetime('now'),
            datetime('now'),
            '$severity',
            '$subject',
            '$message',
            0
        );"
}

# Function to update firewall rules
update_firewall_rules() {
    local ip="$1"
    
    echo "Updating firewall rules for IP: $ip"
    
    # Update CloudPanel's firewall rules
    sqlite3 "$DB_PATH" "
        INSERT INTO firewall_rule (
            created_at,
            updated_at,
            port_range,
            source,
            description
        ) VALUES (
            datetime('now'),
            datetime('now'),
            'ALL',
            '$ip',
            'Automatically blocked due to security incident'
        );"
    
    # Apply firewall rules
    ufw deny from "$ip" to any comment "Auto-blocked by incident response"
}

# Function to initiate security scans
initiate_security_scan() {
    local target_path="$1"
    
    echo "Initiating security scan for: $target_path"
    
    # Record scan initiation
    sqlite3 "$DB_PATH" "
        INSERT INTO event (
            created_at,
            user_name,
            event_name,
            event_data
        ) VALUES (
            datetime('now'),
            'system',
            'security_scan_initiated',
            '{\"path\": \"$target_path\"}'
        );"
    
    # Schedule immediate malware scan
    clamscan -r "$target_path" --move="${CP_HOME}/quarantine/malware"
}

# Function to generate incident reports
generate_incident_report() {
    local report_file="${INCIDENT_DIR}/incident_report_$(date +%Y%m%d).txt"
    
    echo "Generating incident report..."
    
    {
        echo "CloudPanel Incident Report - $(date)"
        echo "================================="
        
        echo -e "\nSecurity Incidents Summary:"
        sqlite3 "$DB_PATH" "
            SELECT 
                event_name,
                COUNT(*) as count,
                MIN(created_at) as first_seen,
                MAX(created_at) as last_seen
            FROM event
            WHERE created_at > datetime('now', '-24 hours')
            AND event_name IN (
                'login_failed',
                'unauthorized_access',
                'file_modification',
                'malware_detected'
            )
            GROUP BY event_name;"
        
        echo -e "\nBlocked IPs:"
        sqlite3 "$DB_PATH" "
            SELECT ip, created_at 
            FROM blocked_ip 
            WHERE created_at > datetime('now', '-24 hours');"
        
        echo -e "\nQuarantined Files:"
        find "${CP_HOME}/quarantine" -type f -mtime -1 -ls
        
    } > "$report_file"
}

# Main execution
main() {
    echo "Starting incident response system..."
    
    while true; do
        # Detect and handle incidents
        detect_security_incidents
        
        # Generate daily incident report at midnight
        if [ "$(date +%H:%M)" = "00:00" ]; then
            generate_incident_report
        fi
        
        # Sleep for monitoring interval
        sleep 300
    done
}

main "$@"

This incident response system provides:

Automated detection and response to security incidents using CloudPanel's event system
Integration with CloudPanel's notification and security tables
Automatic quarantine of suspicious files and malware
IP blocking for brute force attempts
Detailed incident reporting and logging
Automated security scanning when incidents are detected

To implement this incident response system:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-incident-response.sh

# Run the script as a service
sudo systemctl enable cloudpanel-incident-response
sudo systemctl start cloudpanel-incident-response

Securing CloudPanel on Ubuntu 24.04 Part 11

Fri, 22 Nov 2024 00:00:00 GMT

Part 11: Automated Backup and Disaster Recovery System

Let's create a comprehensive backup and recovery system that ensures both data integrity and quick recovery capabilities.

#!/bin/bash

# Create backup management script
sudo nano /usr/local/bin/cloudpanel-backup-manager.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
BACKUP_ROOT="/backup/cloudpanel"
LOG_DIR="${CP_HOME}/logs/backup"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# Create necessary directories
mkdir -p "${BACKUP_ROOT}/{system,sites,databases,ssl,logs}" "$LOG_DIR"

# Function to read site configurations from CloudPanel database
get_site_configurations() {
    sqlite3 "$DB_PATH" "
        SELECT 
            s.id,
            s.domain_name,
            s.user,
            s.root_directory,
            d.name as db_name
        FROM site s
        LEFT JOIN database d ON s.id = d.site_id
        WHERE s.type = 'vhost';"
}

# Function to backup CloudPanel system files
backup_system_files() {
    local backup_dir="${BACKUP_ROOT}/system/${TIMESTAMP}"
    echo "Backing up system files to $backup_dir"
    
    # Create backup directories
    mkdir -p "$backup_dir"/{config,services,database,ssl}
    
    # Backup core configuration
    tar -czf "$backup_dir/config/cloudpanel_config.tar.gz" \
        "${CP_HOME}/services/nginx/nginx.conf" \
        "${CP_HOME}/services/php-fpm" \
        "${CP_HOME}/services/nginx/sites-enabled" \
        --exclude='*.log'
    
    # Backup SSL certificates
    cp -r "${CP_HOME}/services/nginx/ssl-certificates" "$backup_dir/ssl/"
    
    # Backup CloudPanel database
    sqlite3 "$DB_PATH" ".backup '${backup_dir}/database/cloudpanel.sq3'"
    
    # Create backup manifest
    {
        echo "CloudPanel System Backup - $(date)"
        echo "================================="
        echo "Backup Type: System Files"
        echo "Timestamp: $TIMESTAMP"
        find "$backup_dir" -type f -exec ls -lh {} \;
    } > "$backup_dir/MANIFEST.txt"
}

# Function to backup individual sites
backup_site() {
    local domain="$1"
    local user="$2"
    local root_dir="$3"
    local db_name="$4"
    
    local site_backup_dir="${BACKUP_ROOT}/sites/${domain}/${TIMESTAMP}"
    echo "Backing up site: $domain to $site_backup_dir"
    
    # Create site backup directory
    mkdir -p "$site_backup_dir"/{files,database,logs,config}
    
    # Backup site files
    tar -czf "$site_backup_dir/files/site_files.tar.gz" \
        "$root_dir" \
        --exclude='*/cache/*' \
        --exclude='*/tmp/*' \
        --exclude='*/logs/*'
    
    # Backup site database if exists
    if [ ! -z "$db_name" ]; then
        mysqldump --single-transaction \
            --quick \
            --lock-tables=false \
            "$db_name" > "$site_backup_dir/database/${db_name}.sql"
        
        # Compress database backup
        gzip "$site_backup_dir/database/${db_name}.sql"
    fi
    
    # Backup site configuration
    cp "/home/clp/services/nginx/sites-enabled/${domain}.conf" \
        "$site_backup_dir/config/"
    
    # Create site backup manifest
    {
        echo "Site Backup - $domain"
        echo "================================="
        echo "Backup Type: Site Files and Configuration"
        echo "Timestamp: $TIMESTAMP"
        echo "User: $user"
        echo "Database: $db_name"
        find "$site_backup_dir" -type f -exec ls -lh {} \;
    } > "$site_backup_dir/MANIFEST.txt"
}

# Function to verify backup integrity
verify_backup() {
    local backup_dir="$1"
    local backup_type="$2"
    local verification_log="${LOG_DIR}/verification_${TIMESTAMP}.log"
    
    echo "Verifying backup integrity for $backup_type"
    
    # Verify tar archives
    find "$backup_dir" -name "*.tar.gz" -type f | while read -r archive; do
        if ! tar -tzf "$archive" >/dev/null 2>&1; then
            echo "ERROR: Corrupt archive detected: $archive" >> "$verification_log"
            return 1
        fi
    done
    
    # Verify database backups
    find "$backup_dir" -name "*.sql.gz" -type f | while read -r db_backup; do
        if ! gunzip -t "$db_backup" >/dev/null 2>&1; then
            echo "ERROR: Corrupt database backup detected: $db_backup" >> "$verification_log"
            return 1
        fi
    done
    
    # Record successful verification
    echo "Backup verification completed successfully for $backup_type" >> "$verification_log"
    return 0
}

# Function to handle backup rotation
rotate_backups() {
    local backup_path="$1"
    local retention_days=7
    
    echo "Rotating old backups in $backup_path"
    
    # Remove backups older than retention period
    find "$backup_path" -type d -mtime +$retention_days -exec rm -rf {} \;
}

# Function to create recovery script
create_recovery_script() {
    local backup_dir="$1"
    local recovery_script="${backup_dir}/recover.sh"
    
    cat > "$recovery_script" <<'EOF'
#!/bin/bash

# CloudPanel Recovery Script
CP_HOME="/home/clp"

# Verify environment
if [ ! -d "$CP_HOME" ]; then
    echo "ERROR: CloudPanel directory not found!"
    exit 1
fi

# Restore system files
restore_system_files() {
    echo "Restoring system files..."
    tar -xzf config/cloudpanel_config.tar.gz -C /
    
    # Restore SSL certificates
    cp -r ssl/* "${CP_HOME}/services/nginx/ssl-certificates/"
    
    # Restore database
    sqlite3 "${CP_HOME}/htdocs/app/data/db.sq3" ".restore 'database/cloudpanel.sq3'"
}

# Restore site data
restore_site() {
    local site_dir="$1"
    
    echo "Restoring site from: $site_dir"
    
    # Restore site files
    tar -xzf "${site_dir}/files/site_files.tar.gz" -C /
    
    # Restore database if exists
    for db_backup in "${site_dir}"/database/*.sql.gz; do
        if [ -f "$db_backup" ]; then
            local db_name=$(basename "$db_backup" .sql.gz)
            gunzip -c "$db_backup" | mysql "$db_name"
        fi
    done
    
    # Restore configuration
    cp "${site_dir}"/config/* "${CP_HOME}/services/nginx/sites-enabled/"
}

# Main recovery process
main() {
    echo "Starting CloudPanel recovery process..."
    
    # Restore system components
    restore_system_files
    
    # Restore individual sites
    for site_backup in sites/*/${TIMESTAMP}; do
        if [ -d "$site_backup" ]; then
            restore_site "$site_backup"
        fi
    done
    
    # Restart services
    systemctl restart nginx php-fpm mysql
    
    echo "Recovery process completed"
}

main "$@"
EOF

    chmod +x "$recovery_script"
}

# Main execution
main() {
    echo "Starting CloudPanel backup process..."
    
    # Backup system files
    backup_system_files
    verify_backup "${BACKUP_ROOT}/system/${TIMESTAMP}" "system"
    
    # Backup individual sites
    while IFS='|' read -r site_id domain user root_dir db_name; do
        backup_site "$domain" "$user" "$root_dir" "$db_name"
        verify_backup "${BACKUP_ROOT}/sites/${domain}/${TIMESTAMP}" "site"
    done < <(get_site_configurations)
    
    # Create recovery script
    create_recovery_script "${BACKUP_ROOT}/system/${TIMESTAMP}"
    
    # Rotate old backups
    rotate_backups "$BACKUP_ROOT"
    
    echo "Backup process completed successfully"
}

main "$@"

This backup system provides:

Comprehensive backup of CloudPanel system files
Site-specific backups including files, databases, and configurations
Backup verification and integrity checking
Automated backup rotation
Recovery script generation
Integration with CloudPanel's database structure

To implement this backup system:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-backup-manager.sh

# Add to crontab for daily execution
(crontab -l 2>/dev/null; echo "0 2 * * * /usr/local/bin/cloudpanel-backup-manager.sh") | crontab -

Securing CloudPanel on Ubuntu 24.04 Part 12

Sat, 23 Nov 2024 00:00:00 GMT

Part 12: Automated Updates and Patch Management

This system will manage both CloudPanel-specific updates and system-level security patches while maintaining proper logging in CloudPanel.

# Create update management script
sudo nano /usr/local/bin/cloudpanel-update-manager.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
LOG_DIR="${CP_HOME}/logs/updates"
BACKUP_DIR="${CP_HOME}/backups/updates"

mkdir -p "$LOG_DIR" "$BACKUP_DIR"

# Function to get current CloudPanel version
get_cloudpanel_version() {
    sqlite3 "$DB_PATH" "SELECT value FROM config WHERE key = 'app_version';"
}

# Function to check system package updates
check_system_updates() {
    echo "Checking system updates..."
    
    # Update package lists
    apt-get update -qq
    
    # Get list of security updates
    local security_updates=$(apt-get -s upgrade | grep -i security | wc -l)
    
    if [ "$security_updates" -gt 0 ]; then
        log_update_event "security_updates_available" "Security updates available: $security_updates packages"
        return 0
    fi
    return 1
}

# Function to verify PHP versions
check_php_versions() {
    echo "Checking PHP versions..."
    
    # Get installed PHP versions from database
    sqlite3 "$DB_PATH" "
        SELECT DISTINCT php_version 
        FROM php_settings 
        WHERE site_id IN (
            SELECT id FROM site WHERE type = 'vhost'
        );" | while read -r version; do
        
        # Check if this PHP version needs updates
        if php_needs_update "$version"; then
            log_update_event "php_update_required" "PHP version $version needs security updates"
        fi
    done
}

# Function to backup before updates
backup_before_update() {
    local update_type="$1"
    local timestamp=$(date +%Y%m%d_%H%M%S)
    local backup_path="${BACKUP_DIR}/${update_type}_${timestamp}"
    
    echo "Creating pre-update backup..."
    
    # Create backup directory
    mkdir -p "$backup_path"
    
    # Backup CloudPanel database
    sqlite3 "$DB_PATH" ".backup '${backup_path}/cloudpanel.sq3'"
    
    # Backup configuration files
    tar -czf "${backup_path}/config.tar.gz" \
        "${CP_HOME}/services/nginx/nginx.conf" \
        "${CP_HOME}/services/php-fpm" \
        "${CP_HOME}/services/nginx/sites-enabled" \
        --exclude='*.log'
    
    echo "$backup_path"
}

# Function to apply system updates
apply_system_updates() {
    echo "Applying system updates..."
    
    # Create pre-update backup
    local backup_path=$(backup_before_update "system")
    
    # Apply updates
    if DEBIAN_FRONTEND=noninteractive apt-get -y upgrade; then
        log_update_event "system_update_success" "System updates applied successfully"
    else
        log_update_event "system_update_failed" "System update failed"
        restore_from_backup "$backup_path"
        return 1
    fi
}

# Function to update PHP versions
update_php_version() {
    local version="$1"
    echo "Updating PHP version $version..."
    
    # Create backup
    local backup_path=$(backup_before_update "php_${version}")
    
    # Update PHP packages
    if apt-get install -y "php${version}-fpm" "php${version}-common" "php${version}-cli"; then
        log_update_event "php_update_success" "PHP $version updated successfully"
        
        # Restart PHP-FPM service
        systemctl restart "php${version}-fpm"
    else
        log_update_event "php_update_failed" "PHP $version update failed"
        restore_from_backup "$backup_path"
        return 1
    fi
}

# Function to verify system integrity after updates
verify_system_integrity() {
    echo "Verifying system integrity..."
    local integrity_check=0
    
    # Check critical services
    local services=("nginx" "mysql")
    for service in "${services[@]}"; do
        if ! systemctl is-active --quiet "$service"; then
            log_update_event "service_check_failed" "Service $service is not running after update"
            integrity_check=1
        fi
    done
    
    # Check PHP-FPM pools
    sqlite3 "$DB_PATH" "
        SELECT DISTINCT php_version 
        FROM php_settings;" | while read -r version; do
        if ! systemctl is-active --quiet "php${version}-fpm"; then
            log_update_event "php_check_failed" "PHP-FPM $version is not running after update"
            integrity_check=1
        fi
    done
    
    # Verify CloudPanel database
    if ! sqlite3 "$DB_PATH" "PRAGMA integrity_check;"; then
        log_update_event "database_check_failed" "Database integrity check failed after update"
        integrity_check=1
    fi
    
    return $integrity_check
}

# Function to restore from backup
restore_from_backup() {
    local backup_path="$1"
    echo "Restoring from backup: $backup_path"
    
    # Restore CloudPanel database
    sqlite3 "$DB_PATH" ".restore '${backup_path}/cloudpanel.sq3'"
    
    # Restore configuration files
    tar -xzf "${backup_path}/config.tar.gz" -C /
    
    # Restart services
    systemctl restart nginx php-fpm mysql
    
    log_update_event "system_restored" "System restored from backup after failed update"
}

# Function to log update events in CloudPanel's database
log_update_event() {
    local event_name="$1"
    local event_data="$2"
    
    sqlite3 "$DB_PATH" "
        INSERT INTO event (
            created_at,
            user_name,
            event_name,
            event_data
        ) VALUES (
            datetime('now'),
            'system',
            'update_$event_name',
            '$event_data'
        );"
}

# Function to create update report
create_update_report() {
    local report_file="${LOG_DIR}/update_report_$(date +%Y%m%d).txt"
    
    {
        echo "CloudPanel Update Report - $(date)"
        echo "================================="
        
        echo -e "\nSystem Updates:"
        sqlite3 "$DB_PATH" "
            SELECT created_at, event_data 
            FROM event 
            WHERE event_name LIKE 'update_%'
            AND created_at > datetime('now', '-24 hours')
            ORDER BY created_at DESC;"
        
        echo -e "\nInstalled PHP Versions:"
        sqlite3 "$DB_PATH" "
            SELECT DISTINCT php_version, COUNT(*) as sites
            FROM php_settings
            GROUP BY php_version;"
        
        echo -e "\nService Status:"
        systemctl status nginx mysql | grep Active
        
    } > "$report_file"
}

# Main execution
main() {
    echo "Starting update management system..."
    
    # Check for updates
    if check_system_updates; then
        # Apply system updates
        apply_system_updates
    fi
    
    # Check PHP versions
    check_php_versions
    
    # Verify system integrity
    if ! verify_system_integrity; then
        log_update_event "integrity_check_failed" "System integrity check failed after updates"
    fi
    
    # Create update report
    create_update_report
    
    echo "Update management completed"
}

main "$@"

This update management system provides:

Automated system updates with pre-update backups
PHP version management integrated with CloudPanel's configuration
System integrity verification
Automatic rollback capabilities
Detailed update logging in CloudPanel's event system
Update reporting and monitoring

To implement this update management system:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-update-manager.sh

# Add to crontab for weekly execution
(crontab -l 2>/dev/null; echo "0 3 * * 0 /usr/local/bin/cloudpanel-update-manager.sh") | crontab -

Securing CloudPanel on Ubuntu 24.04 Part 2

Thu, 14 Nov 2024 00:00:00 GMT

Part 2: Service Security and Advanced Monitoring

1. Service Security Integration

Let's implement comprehensive service monitoring:

# Create service monitoring script
sudo nano /usr/local/bin/cloudpanel-service-monitor.sh

#!/bin/bash

set -euo pipefail

# Configuration
ENABLE_EMAIL="false"
ENABLE_DISCORD="true"
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/your/yourapi"
GMAIL_USER="your-gmail@gmail.com"
GMAIL_APP_PASSWORD="your-app-password"
EMAIL_TO="your-org-email@domain.com"

# CloudPanel specific paths
CP_HOME="/home/clp"
CP_DB="${CP_HOME}/htdocs/app/data/db.sq3"
SITES_ROOT="/home"

# Thresholds
CPU_THRESHOLD=80
MEMORY_THRESHOLD=90
PROCESS_THRESHOLD=500
DISK_THRESHOLD=90
ERROR_LOG_THRESHOLD=$((1024*1024*10))  # 10MB
HTTP_ERROR_THRESHOLD=100
CHECK_INTERVAL=300

# Setup
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
OUTPUT_DIR="/var/log/cloudpanel_monitoring"
ALERT_LOG="${OUTPUT_DIR}/alerts.log"
mkdir -p "${OUTPUT_DIR}"

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

log() {
    echo -e "${2}[$(date '+%Y-%m-%d %H:%M:%S')] ${1}${NC}" | tee -a "${OUTPUT_DIR}/monitoring.log"
}

log_event() {
    local event_name="$1"
    local event_data="$2"
    local severity="${3:-normal}"
    
    # Log to CloudPanel database
    sqlite3 "$CP_DB" "
        INSERT INTO event (
            created_at,
            user_name,
            user_role,
            event_name,
            event_data,
            source_ip_address,
            user_agent
        ) VALUES (
            datetime('now'),
            'system',
            'ROLE_ADMIN',
            '$event_name',
            '$event_data',
            'localhost',
            'system-monitor'
        );"
    
    # Create notification if severity is high
    if [ "$severity" = "high" ]; then
        # Generate a unique hash for the notification
        local hash=$(echo "${event_name}${event_data}$(date +%s)" | md5sum | cut -d' ' -f1)
        
        sqlite3 "$CP_DB" "
            INSERT INTO notification (
                created_at,
                updated_at,
                hash,
                severity,
                subject,
                message,
                is_read
            ) VALUES (
                datetime('now'),
                datetime('now'),
                '$hash',
                'high',
                '$event_name',
                '$event_data',
                0
            );"
    fi
}

validate_config() {
    local has_error=false

    if [ "$ENABLE_EMAIL" = "true" ]; then
        if [ -z "$GMAIL_USER" ] || [ "$GMAIL_USER" = "your-gmail@gmail.com" ]; then
            log "Error: GMAIL_USER not configured" "${RED}"
            has_error=true
        fi
        if [ -z "$GMAIL_APP_PASSWORD" ] || [ "$GMAIL_APP_PASSWORD" = "your-app-password" ]; then
            log "Error: GMAIL_APP_PASSWORD not configured" "${RED}"
            has_error=true
        fi
        if [ -z "$EMAIL_TO" ] || [ "$EMAIL_TO" = "your-org-email@domain.com" ]; then
            log "Error: EMAIL_TO not configured" "${RED}"
            has_error=true
        fi
    fi

    if [ "$ENABLE_DISCORD" = "true" ]; then
        if [ -z "$DISCORD_WEBHOOK_URL" ] || \
           [ "$DISCORD_WEBHOOK_URL" = "https://discord.com/api/webhooks/your/yourapi" ] || \
           ! echo "$DISCORD_WEBHOOK_URL" | grep -q "^https://discord.com/api/webhooks/[0-9]*/"; then
            log "Error: DISCORD_WEBHOOK_URL not properly configured" "${RED}"
            log "Please set ENABLE_DISCORD=false or configure a valid webhook URL" "${RED}"
            ENABLE_DISCORD=false  # Disable Discord if URL is invalid
        fi
    fi

    if [ "$has_error" = "true" ]; then
        log "Configuration validation failed. Please check settings." "${RED}"
        exit 1
    fi
}

send_discord_alert() {
    local title="$1"
    local message="$2"
    local priority="$3"

    local escaped_title=$(echo "$title" | sed 's/"/\\"/g')
    local escaped_message=$(echo "$message" | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')
    local escaped_hostname=$(hostname | sed 's/"/\\"/g')

    local color
    if [ "$priority" = "high" ]; then
        color="16711680"
    else
        color="15844367"
    fi

    local json_data
    printf -v json_data '{
        "embeds": [{
            "title": "%s",
            "description": "%s",
            "color": %d,
            "footer": {
                "text": "CloudPanel Monitoring - %s"
            },
            "timestamp": "%s"
        }]
    }' "$escaped_title" "$escaped_message" "$color" "$escaped_hostname" "$(date -u +%Y-%m-%dT%H:%M:%SZ)"

    curl -s -H "Content-Type: application/json" -X POST -d "${json_data}" "${DISCORD_WEBHOOK_URL}"
}

send_gmail() {
    local subject="$1"
    local message="$2"
    local priority="$3"

    local email_content=$(mktemp)
    
    cat > "${email_content}" << EOF
From: ${GMAIL_USER}
To: ${EMAIL_TO}
Subject: $([ "$priority" = "high" ] && echo "[URGENT] ")${subject}
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"

${message}
EOF

    curl --url "smtps://smtp.gmail.com:465" \
         --ssl-reqd \
         --mail-from "${GMAIL_USER}" \
         --mail-rcpt "${EMAIL_TO}" \
         --upload-file "${email_content}" \
         --user "${GMAIL_USER}:${GMAIL_APP_PASSWORD}" \
         --silent

    rm -f "${email_content}"
}

send_alert() {
    local subject="$1"
    local message="$2"
    local priority="$3"
    
    local formatted_message="
Server: $(hostname)
Time: $(date)

${message}

This is an automated alert from CloudPanel Monitoring System."
    
    if [ "$ENABLE_EMAIL" = "true" ]; then
        send_gmail "${subject}" "${formatted_message}" "${priority}"
        log "Email alert sent: ${subject}" "${YELLOW}"
    fi
    
    if [ "$ENABLE_DISCORD" = "true" ]; then
        send_discord_alert "${subject}" "${formatted_message}" "${priority}"
        log "Discord alert sent: ${subject}" "${YELLOW}"
    fi
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] ${subject}" >> "${ALERT_LOG}"
    log_event "system_alert" "${subject}: ${message}" "${priority}"
}

check_system_resources() {
    log "Checking system resources..." "${BLUE}"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # CPU Usage
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d. -f1)
    sqlite3 "$CP_DB" "INSERT INTO instance_cpu (created_at, value) VALUES ('$timestamp', $cpu_usage);"
    
    if [ "${cpu_usage:-0}" -gt "$CPU_THRESHOLD" ]; then
        local top_processes=$(ps aux --sort=-%cpu | head -n 6)
        send_alert "High CPU Usage Alert" "CPU Usage: ${cpu_usage}%\n\nTop Processes:\n${top_processes}" "high"
    fi
    
    # Memory Usage
    local memory_usage=$(free | grep Mem | awk '{print ($3/$2 * 100)}' | cut -d. -f1)
    sqlite3 "$CP_DB" "INSERT INTO instance_memory (created_at, value) VALUES ('$timestamp', $memory_usage);"
    
    if [ "${memory_usage:-0}" -gt "$MEMORY_THRESHOLD" ]; then
        local memory_processes=$(ps aux --sort=-%mem | head -n 6)
        send_alert "High Memory Usage Alert" "Memory Usage: ${memory_usage}%\n\nTop Memory Processes:\n${memory_processes}" "high"
    fi
    
    # Load Average
    # Read and clean load average values
    read -r load1 load5 load15 _ _ < /proc/loadavg
    
    # Convert to float values and remove any extra content
    load1=$(printf "%.2f" "${load1%%/*}")
    load5=$(printf "%.2f" "${load5%%/*}")
    load15=$(printf "%.2f" "${load15%%/*}")
    
    # Insert cleaned values into database
    sqlite3 "$CP_DB" "
        INSERT INTO instance_load_average (created_at, period, value) 
        VALUES 
            ('$timestamp', 1, $load1),
            ('$timestamp', 5, $load5),
            ('$timestamp', 15, $load15);"
    
    # Process Count
    local process_count=$(ps aux | wc -l)
    if [ "${process_count:-0}" -gt "$PROCESS_THRESHOLD" ]; then
        send_alert "High Process Count Alert" "Process Count: ${process_count}\n\nProcess List:\n$(ps aux | head -n 6)" "high"
    fi
    
    # Disk Usage
    df -h | grep '^/dev/' | while read -r device size used avail use_percent mount; do
        local usage=${use_percent%\%}
        sqlite3 "$CP_DB" "INSERT INTO instance_disk_usage (created_at, disk, value) VALUES ('$timestamp', '$mount', $usage);"
        
        if [ "${usage:-0}" -gt "$DISK_THRESHOLD" ]; then
            local disk_info=$(df -h)
            send_alert "High Disk Usage Alert" "Disk Usage: ${usage}%\n\nDisk Information:\n${disk_info}" "high"
        fi
    done
}

check_services() {
    log "Checking critical services..." "${BLUE}"
    
    # Check basic services
    local basic_services=("nginx" "mysql" "varnish" "redis-server")
    for service in "${basic_services[@]}"; do
        if ! systemctl is-active --quiet "$service"; then
            send_alert "Service Down Alert" "Service ${service} is not running!" "high"
            log_event "service_down" "Service ${service} is not running" "high"
        fi
    done
    
    # Check PHP-FPM services from database
    local php_versions=$(sqlite3 "$CP_DB" "SELECT DISTINCT php_version FROM php_settings WHERE php_version IS NOT NULL;")
    for version in $php_versions; do
        local service_name="php${version}-fpm"
        if systemctl list-unit-files "${service_name}.service" &>/dev/null; then
            if ! systemctl is-active --quiet "$service_name"; then
                local status_details=$(systemctl status "${service_name}" 2>&1 | head -n 4)
                send_alert "PHP-FPM Service Issue" "PHP ${version} FPM service is down!\n\nStatus Details:\n${status_details}" "high"
                log_event "php_fpm_down" "PHP-FPM ${version} is not running" "high"
            fi
        fi
    done
}

check_website_resources() {
    log "Checking website resources..." "${BLUE}"
    
    # Query all active sites from database
    sqlite3 "$CP_DB" "SELECT id, domain_name, user FROM site WHERE type = 'vhost';" | while IFS='|' read -r site_id domain user; do
        # Skip if user directory doesn't exist
        if [ ! -d "${SITES_ROOT}/${user}" ]; then
            continue
        fi
        
        # Check nginx logs
        local nginx_log_dir="${SITES_ROOT}/${user}/logs/nginx"
        local access_log="${nginx_log_dir}/${domain}-access.log"
        local error_log="${nginx_log_dir}/${domain}-error.log"
        
        if [ -f "$access_log" ]; then
            # Check for high traffic IPs
            local high_traffic_ips=$(tail -n 1000 "$access_log" | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 5)
            local top_ip_count=$(echo "$high_traffic_ips" | head -n 1 | awk '{print $1}')
            
            if [ "${top_ip_count:-0}" -gt 500 ]; then
                send_alert "High Traffic Alert" "High traffic detected for ${domain}:\n${high_traffic_ips}" "normal"
                log_event "high_traffic" "High traffic on ${domain}" "normal"
            fi
            
            # Check for high error rates
            local error_count=$(tail -n 1000 "$access_log" | awk '$9 ~ /^[45]/ {print $9}' | wc -l)
            if [ "${error_count:-0}" -gt "$HTTP_ERROR_THRESHOLD" ]; then
                local error_details=$(tail -n 1000 "$access_log" | awk '$9 ~ /^[45]/ {print $1,$9,$7}' | tail -n 10)
                send_alert "High Error Rate Alert" "Domain: ${domain}\nError count: ${error_count}\n\nLast errors:\n${error_details}" "high"
                log_event "high_error_rate" "High error rate on ${domain}: ${error_count} errors" "high"
            fi
        fi
        
        # Check PHP logs
        local php_log_dir="${SITES_ROOT}/${user}/logs/php"
        local php_error_log="${php_log_dir}/error.log"
        if [ -f "$php_error_log" ]; then
            local log_size=$(stat -c %s "$php_error_log")
            if [ "${log_size:-0}" -gt "$ERROR_LOG_THRESHOLD" ]; then
                send_alert "Large PHP Error Log" "Domain: ${domain}\nPHP error log size: $(numfmt --to=iec-i --suffix=B ${log_size})" "high"
                log_event "large_php_log" "Large PHP error log for ${domain}" "high"
            fi
            
            # Check for recent fatal errors
            local fatal_errors=$(tail -n 1000 "$php_error_log" | grep -i "fatal error" | tail -n 5)
            if [ -n "$fatal_errors" ]; then
                send_alert "PHP Fatal Errors Detected" "Domain: ${domain}\n\nRecent fatal errors:\n${fatal_errors}" "high"
                log_event "php_fatal_errors" "PHP fatal errors detected on ${domain}" "high"
            fi
        fi
        
        # Check varnish cache if enabled
        if [ -d "${SITES_ROOT}/${user}/.varnish-cache" ]; then
            local varnish_log="${SITES_ROOT}/${user}/logs/varnish-cache/purge.log"
            if [ -f "$varnish_log" ]; then
                local purge_count=$(tail -n 1000 "$varnish_log" | wc -l)
                if [ "${purge_count:-0}" -gt 1000 ]; then
                    send_alert "High Varnish Purge Rate" "Domain: ${domain}\nHigh number of cache purges: ${purge_count} in recent log" "normal"
                    log_event "high_purge_rate" "High varnish purge rate on ${domain}" "normal"
                fi
            fi
        fi
        
        # Check application-specific settings
        local app_type=$(sqlite3 "$CP_DB" "SELECT application FROM site WHERE id = $site_id;")
        case "$app_type" in
            "wordpress")
                local wp_config="${SITES_ROOT}/${user}/htdocs/${domain}/wp-config.php"
                if [ -f "$wp_config" ]; then
                    if grep -q "WP_DEBUG.*true" "$wp_config"; then
                        send_alert "WordPress Debug Enabled" "Domain: ${domain}\nWP_DEBUG is enabled in production" "normal"
                        log_event "wp_debug_enabled" "WordPress debug mode enabled on ${domain}" "normal"
                    fi
                fi
                ;;
        esac
    done
}

check_security() {
    log "Checking security..." "${BLUE}"
    
    # Check failed SSH attempts
    local failed_ssh=$(grep "Failed password" /var/log/auth.log 2>/dev/null | awk '{print $11}' | sort | uniq -c | sort -nr | head -n 5)
    if [ -n "$failed_ssh" ]; then
        send_alert "Security Alert - Failed SSH Attempts" "Recent failed SSH attempts:\n${failed_ssh}" "high"
        log_event "failed_ssh_attempts" "Multiple failed SSH attempts detected" "high"
    fi
    
    # Check for modified system files
    local modified_files=$(find /etc /usr/bin /usr/sbin -mmin -60 -type f 2>/dev/null)
    if [ -n "$modified_files" ]; then
        send_alert "Security Alert - Modified System Files" "Recently modified system files:\n${modified_files}" "high"
        log_event "modified_system_files" "System files modified in last hour" "high"
    fi
    
    # Check CloudPanel specific security
    local blocked_ips=$(sqlite3 "$CP_DB" "SELECT COUNT(*) FROM blocked_ip;")
    local blocked_bots=$(sqlite3 "$CP_DB" "SELECT COUNT(*) FROM blocked_bot;")
    
    if [ "${blocked_ips:-0}" -gt 0 ] || [ "${blocked_bots:-0}" -gt 0 ]; then
        send_alert "Security Info - Blocked Entities" "Current blocks:\nIPs: ${blocked_ips}\nBots: ${blocked_bots}" "normal"
    fi
    
    # Check for expired SSL certificates
    local near_expiry=$(sqlite3 "$CP_DB" "
        SELECT s.domain_name, c.expires_at 
        FROM certificate c 
        JOIN site s ON c.site_id = s.id 
        WHERE c.expires_at <= datetime('now', '+30 days') 
        AND c.expires_at > datetime('now');"
    )
    
    if [ -n "$near_expiry" ]; then
        send_alert "SSL Certificates Near Expiry" "Certificates expiring soon:\n${near_expiry}" "normal"
        log_event "ssl_near_expiry" "SSL certificates approaching expiration" "normal"
    fi
}

check_scheduled_tasks() {
    log "Checking scheduled tasks..." "${BLUE}"
    
    # Check cron jobs from database
    local failed_crons=$(sqlite3 "$CP_DB" "
        SELECT s.domain_name, cj.command 
        FROM cron_job cj 
        JOIN site s ON cj.site_id = s.id;"
    )
    
    if [ -n "$failed_crons" ]; then
        while IFS='|' read -r domain command; do
            # Basic validation of cron command existence
            local cmd_path=$(echo "$command" | awk '{print $1}')
            if [ ! -f "$cmd_path" ] && ! command -v "$cmd_path" >/dev/null 2>&1; then
                send_alert "Invalid Cron Job" "Domain: ${domain}\nInvalid command path: ${cmd_path}" "normal"
                log_event "invalid_cron" "Invalid cron command for ${domain}" "normal"
            fi
        done <<< "$failed_crons"
    fi
}

main() {
    log "Starting CloudPanel monitoring system..." "${GREEN}"
    validate_config
    
    while true; do
        check_system_resources
        check_services
        check_website_resources
        check_security
        check_scheduled_tasks
        
        log "Sleeping for ${CHECK_INTERVAL} seconds..." "${BLUE}"
        sleep "${CHECK_INTERVAL}"
    done
}

main "$@"

This script:

Uses CloudPanel's actual database schema for storing events and metrics
Monitors all services configured in the database
Records system metrics in CloudPanel's instance_* tables
Monitors website resources based on CloudPanel's site structure
Maintains proper logging in CloudPanel's event system

To implement:

# Make script executable
chmod +x /usr/local/bin/cloudpanel-service-monitor.sh

# Create systemd service
sudo nano /etc/systemd/system/cloudpanel-monitor.service

[Unit]
Description=CloudPanel Service Monitor
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/cloudpanel-service-monitor.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

# Enable and start the service
sudo systemctl enable cloudpanel-monitor
sudo systemctl start cloudpanel-monitor

Securing CloudPanel on Ubuntu 24.04 Part 3

Fri, 15 Nov 2024 00:00:00 GMT

Part 3: Web Security and Application-Level Protection

I'll explain how to implement comprehensive web security measures specifically tailored for CloudPanel. Let's break this down into manageable components that work together to create a robust security layer.

1. Enhanced Nginx Security Configuration

First, let's create a security-focused Nginx configuration that aligns with CloudPanel. CloudPanel uses a dual Nginx configuration approach, with core settings in /home/clp/services/nginx/ and site-specific configurations in /etc/nginx/:

# Create a security-focused Nginx configuration
sudo nano /etc/nginx/conf.d/security-headers.conf

Let's implement strong security headers while maintaining CloudPanel's functionality:

# Enhanced Security Headers Configuration
# These headers protect against common web vulnerabilities while ensuring CloudPanel operates correctly

# Basic security headers
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

# Content Security Policy tailored for CloudPanel
add_header Content-Security-Policy "
    default-src 'self';
    script-src 'self' 'unsafe-inline' 'unsafe-eval';
    style-src 'self' 'unsafe-inline';
    img-src 'self' data: https:;
    font-src 'self' data:;
    frame-ancestors 'self';
    form-action 'self';
" always;

# HSTS configuration - Enable only after ensuring SSL is properly configured
# add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# Protection against various attacks
client_body_buffer_size 10K;
client_max_body_size 10M;
client_body_timeout 12;
client_header_timeout 12;

# Rate limiting configuration
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
limit_conn_zone $binary_remote_addr zone=addr:10m;

2. PHP Security Hardening

CloudPanel manages multiple PHP versions, so we need to secure each version while maintaining compatibility. Let's create a comprehensive PHP security configuration:

# Create a function to apply security settings to all PHP versions
sudo nano /usr/local/bin/cloudpanel-php-secure.sh

Here's a script that automatically secures all PHP versions used by CloudPanel:

#!/bin/bash

# CloudPanel Security Manager
# This script implements security measures for CloudPanel installations
# focusing on PHP hardening and firewall configurations.

set -euo pipefail
IFS=$'\n\t'

# Configuration variables
readonly CP_HOME="/home/clp"
readonly SITES_ROOT="/home"
readonly DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
readonly LOG_DIR="/var/log/cloudpanel_monitoring"
readonly LOG_FILE="${LOG_DIR}/security_manager.log"
readonly ALERT_LOG="${LOG_DIR}/alerts.log"
readonly BACKUP_DIR="${LOG_DIR}/backups"

# Color definitions
readonly RED='\033[0;31m'
readonly GREEN='\033[0;32m'
readonly YELLOW='\033[1;33m'
readonly BLUE='\033[0;34m'
readonly NC='\033[0m'

# Security settings
readonly DEFAULT_MEMORY_LIMIT="256M"
readonly DEFAULT_UPLOAD_SIZE="10M"

# Initialize logging
setup_logging() {
    mkdir -p "${LOG_DIR}" "${BACKUP_DIR}"
    
    if [[ ! -f "${LOG_FILE}" ]]; then
        echo "=== CloudPanel Security Manager Log Started $(date '+%Y-%m-%d %H:%M:%S') ===" > "${LOG_FILE}"
    fi
    
    if [[ ! -f "${ALERT_LOG}" ]]; then
        echo "=== CloudPanel Security Alerts Started $(date '+%Y-%m-%d %H:%M:%S') ===" > "${ALERT_LOG}"
    fi
    
    # Set up log rotation
    cat > "/etc/logrotate.d/cloudpanel-security" << 'LOGROTATE'
/var/log/cloudpanel_monitoring/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root root
}
LOGROTATE

    log "INFO" "System" "Logging initialized"
}

# Log function with component tracking
log() {
    local level=$1
    local component=$2
    shift 2
    local message="$*"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Select color based on level
    local color=""
    case "${level}" in
        "ERROR") color="${RED}" ;;
        "WARN")  color="${YELLOW}" ;;
        "INFO")  color="${GREEN}" ;;
        "DEBUG") color="${BLUE}" ;;
    esac
    
    # Log to file
    echo "${timestamp} | ${level} | ${component} | ${message}" >> "${LOG_FILE}"
    
    # Console output
    echo -e "${color}[${timestamp}] [${level}] ${component}: ${message}${NC}"
    
    # Log alerts
    if [[ "${level}" == "ERROR" || "${level}" == "WARN" ]]; then
        echo "${timestamp} | ${level} | ${component} | ${message}" >> "${ALERT_LOG}"
    fi
}

# Backup function
backup_file() {
    local file=$1
    local backup="${BACKUP_DIR}/$(basename "${file}").$(date +%Y%m%d_%H%M%S)"
    
    if [[ -f "${file}" ]]; then
        cp "${file}" "${backup}"
        log "INFO" "Backup" "Created backup: ${backup}"
    else
        log "WARN" "Backup" "File not found: ${file}"
        return 1
    fi
}

# Configure firewall
configure_firewall() {
    log "INFO" "Firewall" "Configuring firewall rules"
    
    sqlite3 "${DB_PATH}" << 'FWEND'
BEGIN TRANSACTION;
DELETE FROM firewall_rule;
INSERT INTO firewall_rule (created_at, updated_at, port_range, source, description)
VALUES 
    (datetime('now'), datetime('now'), '80,443', '0.0.0.0/0', 'Web Traffic'),
    (datetime('now'), datetime('now'), '22', '0.0.0.0/0', 'SSH Access'),
    (datetime('now'), datetime('now'), '8443', '127.0.0.1,::1', 'CloudPanel Local Access');
COMMIT;
FWEND
    
    log "INFO" "Firewall" "Rules updated successfully"
}

# Configure PHP security
configure_php_security() {
    local site_id=$1
    local domain=$2
    
    log "INFO" "PHP" "Configuring security for ${domain}"
    
    sqlite3 "${DB_PATH}" << PHPEND
UPDATE php_settings SET
    memory_limit = '${DEFAULT_MEMORY_LIMIT}',
    max_execution_time = 30,
    max_input_time = 60,
    max_input_vars = 1000,
    post_max_size = '${DEFAULT_UPLOAD_SIZE}',
    upload_max_file_size = '${DEFAULT_UPLOAD_SIZE}',
    additional_configuration = '
expose_php = Off
allow_url_fopen = Off
allow_url_include = Off
session.cookie_httponly = 1
session.cookie_secure = 1
session.cookie_samesite = "Strict"
session.use_strict_mode = 1
display_errors = Off
log_errors = On
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
disable_functions = exec,passthru,shell_exec,system,proc_open,popen
max_execution_time = 30
memory_limit = ${DEFAULT_MEMORY_LIMIT}'
WHERE site_id = ${site_id};
PHPEND

    log "INFO" "PHP" "Settings updated for ${domain}"
}

# Process site
process_site() {
    local site_id=$1
    local domain=$2
    
    log "INFO" "Site" "Processing ${domain}"
    configure_php_security "${site_id}" "${domain}"
}

# Main function
main() {
    local start_time=$(date +%s)
    
    setup_logging
    log "INFO" "System" "Starting security configuration"
    
    backup_file "${DB_PATH}"
    configure_firewall
    
    while IFS='|' read -r site_id domain; do
        process_site "${site_id}" "${domain}"
    done < <(sqlite3 "${DB_PATH}" "SELECT id, domain_name FROM site;")
    
    log "INFO" "System" "Restarting PHP-FPM"
    systemctl restart "php*-fpm"
    
    local end_time=$(date +%s)
    log "INFO" "System" "Completed in $((end_time - start_time)) seconds"
}

# Run main with error handling
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    trap 'log "ERROR" "System" "Failed on line $LINENO"' ERR
    main "$@"
fi

3. Web Application Firewall Integration

Let's implement a WAF configuration that works with CloudPanel's structure:

Install Naxsi WAF for Nginx

Here's a CloudPanel-specific Naxsi WAF configuration:

Examples for CloudPanel

# Include in the server block of your WordPress site
server {
    # Standard CloudPanel configuration remains unchanged
    listen 80;
    listen [::]:80;
    server_name example.com;

    # Enable Naxsi base rules
    include /etc/nginx/naxsi/naxsi-base.conf;

    # WordPress-specific whitelist rules
    include /etc/nginx/naxsi/wordpress.rules;

    location / {
        # Standard WordPress handling
        try_files $uri $uri/ /index.php?$args;

        # WordPress-specific Naxsi configurations
        BasicRule wl:1315,1101 "mz:$BODY_VAR:post_title";
        BasicRule wl:1315,1101 "mz:$BODY_VAR:content";
        BasicRule wl:1315 "mz:$BODY_VAR:excerpt";
        BasicRule wl:1101 "mz:$BODY_VAR:cat";

        # Allow WordPress admin actions
        BasicRule wl:1000,1015 "mz:$URL:/wp-admin/|$BODY_VAR:action";
    }

    # Special handling for wp-admin
    location /wp-admin {
        # Allow file uploads in wp-admin
        BasicRule wl:1500 "mz:$URL:/wp-admin/|$BODY_VAR:attachment";
        BasicRule wl:1310,1311 "mz:$URL:/wp-admin/|BODY";

        # Standard CloudPanel proxy configuration
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Securing CloudPanel on Ubuntu 24.04 Part 4 and 5

Sat, 16 Nov 2024 00:00:00 GMT

Part 4 & 5: Advanced Security Controls and PHP Configuration Management

Let's create a security system that integrates traffic analysis and PHP configuration management based on CloudPanel.

First, let's create our main security management script:

# Create the security management script
sudo nano /usr/local/bin/cloudpanel-php-security-manager.sh

#!/bin/bash

# CloudPanel PHP Security Manager
# Manages PHP-FPM configurations and security settings across different PHP versions

set -euo pipefail

# Core paths based on CloudPanel structure
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
SITES_ROOT="/home"
LOG_DIR="${CP_HOME}/logs/php"

mkdir -p "$LOG_DIR"

# Function to handle logging with timestamps and levels
log() {
    local level="$1"
    local message="$2"
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $message" | tee -a "${LOG_DIR}/security.log"
}

# Validate database existence
validate_db() {
    if [ ! -f "$DB_PATH" ]; then
        log "ERROR" "CloudPanel database not found at $DB_PATH"
        exit 1
    fi
}

# Retrieve sites with their PHP settings from database
# This function joins site and php_settings tables to get complete configuration
get_sites_with_php() {
    sqlite3 "$DB_PATH" "
SELECT 
    s.id, 
    s.domain_name, 
    s.user, 
    s.root_directory, 
    p.php_version, 
    p.id as php_settings_id, 
    p.memory_limit, 
    p.max_execution_time, 
    p.additional_configuration, 
    p.pool_port
FROM site s
INNER JOIN php_settings p ON s.php_settings_id = p.id
WHERE s.type = 'vhost';"
}

# Configure PHP version specific settings
configure_php_version() {
    local php_version="$1"
    local version_number="${php_version/./}"  # Convert 7.4 to 74
    
    # Verify PHP version installation
    if [ ! -d "/etc/php/${php_version}" ]; then
        log "ERROR" "PHP version ${php_version} not installed"
        return 1
    fi
    
    # Setup PHP-FPM pool directory
    local fpm_pool_dir="/etc/php/${php_version}/fpm/pool.d"
    mkdir -p "$fpm_pool_dir"
    
    # Create base PHP-FPM configuration if missing
    local fpm_conf="/etc/php/${php_version}/fpm/php-fpm.conf"
    if [ ! -f "$fpm_conf" ]; then
        cat > "$fpm_conf" << EOF
[global]
pid = /run/php/php${php_version}-fpm.pid
error_log = /var/log/php${version_number}-fpm.log
log_level = notice
emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s
daemonize = yes
include=/etc/php/${php_version}/fpm/pool.d/*.conf
EOF
    fi
    
    # Enable PHP-FPM service
    systemctl enable "php${version_number}-fpm" 2>/dev/null || true
}

# Configure PHP settings for a specific site
configure_site_php() {
    local domain="$1"
    local user="$2"
    local php_version="$3"
    local root_dir="$4"
    local memory_limit="$5"
    local max_exec="$6"
    local additional_config="$7"
    local pool_port="$8"
    
    local version_number="${php_version/./}"
    local pool_config="/etc/php/${php_version}/fpm/pool.d/${domain}.conf"
    local user_php_log_dir="${SITES_ROOT}/${user}/logs/php"
    
    # Create and set permissions for log directory
    mkdir -p "$user_php_log_dir"
    chown "${user}:${user}" "$user_php_log_dir"
    
    # Create PHP-FPM pool configuration
    cat > "$pool_config" << EOF
[${domain}]
user = ${user}
group = ${user}
listen = 127.0.0.1:${pool_port}

; Process Management
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 2
pm.max_spare_servers = 10

; Path Settings
chdir = ${root_dir}

; PHP Settings
php_admin_flag[log_errors] = on
php_admin_value[error_log] = ${user_php_log_dir}/php-${domain}-error.log
php_admin_value[memory_limit] = ${memory_limit}
php_admin_value[max_execution_time] = ${max_exec}
php_admin_value[session.save_handler] = files
php_admin_value[session.save_path] = ${SITES_ROOT}/${user}/tmp/sessions
php_admin_value[upload_tmp_dir] = ${SITES_ROOT}/${user}/tmp/upload

; Security Settings
php_admin_flag[allow_url_fopen] = off
php_admin_flag[allow_url_include] = off
php_admin_value[disable_functions] = exec,passthru,shell_exec,system,proc_open,popen
php_admin_value[open_basedir] = ${root_dir}:/tmp:${SITES_ROOT}/${user}/tmp

; Custom Configuration
${additional_config}
EOF

    # Setup temporary directories with proper permissions
    mkdir -p "${SITES_ROOT}/${user}/tmp/"{sessions,upload}
    chown -R "${user}:${user}" "${SITES_ROOT}/${user}/tmp"
    chmod -R 750 "${SITES_ROOT}/${user}/tmp"

    log "INFO" "Configured PHP-FPM pool for ${domain} (PHP ${php_version})"
}

# Setup log rotation for PHP logs
setup_log_rotation() {
    local user="$1"
    local domain="$2"
    
    cat > "/etc/logrotate.d/php-${domain}" << EOF
${SITES_ROOT}/${user}/logs/php/php-${domain}-error.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 ${user} ${user}
}
EOF
}

# Main execution function
main() {
    log "INFO" "Starting CloudPanel PHP security configuration"
    validate_db
    
    # Create backup directory
    local backup_date=$(date +%Y%m%d_%H%M%S)
    local backup_dir="${CP_HOME}/backups/php_security_${backup_date}"
    mkdir -p "$backup_dir"
    
    # Track configured PHP versions
    declare -A configured_versions
    
    # Process each site
    while IFS='|' read -r site_id domain user root_dir php_version settings_id mem_limit max_exec additional port; do
        if [ -z "${configured_versions[$php_version]:-}" ]; then
            configure_php_version "$php_version"
            configured_versions[$php_version]=1
            
            # Backup existing configuration
            if [ -d "/etc/php/${php_version}" ]; then
                cp -r "/etc/php/${php_version}" "${backup_dir}/php${php_version}"
            fi
        fi
        
        configure_site_php "$domain" "$user" "$php_version" "$root_dir" "$mem_limit" "$max_exec" "$additional" "$port"
        setup_log_rotation "$user" "$domain"
    done < <(get_sites_with_php)
    
    log "INFO" "PHP security configuration completed"
    log "INFO" "Backup stored in: $backup_dir"
    
    # Reload PHP-FPM for each configured version
    for version in "${!configured_versions[@]}"; do
        version_num="${version/./}"
        systemctl reload "php${version_num}-fpm" || log "ERROR" "Failed to reload PHP-FPM for version ${version}"
    done
}

main "$@"

This unified script:

Reads site configurations directly from CloudPanel
Configures PHP-FPM pools with settings from the database
Implements security measures for both PHP
Analyzes traffic patterns for suspicious activity
Creates proper backups before making changes

To use this script:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-php-security-manager.sh

# Run the script
/usr/local/bin/cloudpanel-php-security-manager.sh

Securing CloudPanel on Ubuntu 24.04 Part 6

Sun, 17 Nov 2024 00:00:00 GMT

Part 6: Database Security and Backup Management

Let's create a database security management system:

# Create database security management script
sudo nano /usr/local/bin/cloudpanel-db-security.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
BACKUP_DIR="${CP_HOME}/backups/databases"
LOG_DIR="${CP_HOME}/logs/database"

# Ensure our directories exist
mkdir -p "$BACKUP_DIR" "$LOG_DIR"

# Function to read database configurations from CloudPanel
get_database_config() {
    sqlite3 "$DB_PATH" "
        SELECT 
            d.id,
            d.name as db_name,
            d.site_id,
            s.domain_name,
            ds.engine,
            ds.host,
            ds.port,
            du.user_name,
            du.permissions
        FROM database d
        JOIN database_server ds ON d.database_server_id = ds.id
        JOIN site s ON d.site_id = s.id
        JOIN database_user du ON d.id = du.database_id
        WHERE ds.is_active = 1;"
}

# Function to implement database security measures
secure_databases() {
    echo "Implementing database security measures..."
    
    while IFS='|' read -r db_id db_name site_id domain engine host port user perms; do
        echo "Securing database: $db_name for $domain"
        
        # Create secure configuration for each database
        case "$engine" in
            "mysql")
                secure_mysql_database "$db_name" "$user" "$perms"
                ;;
            "postgresql")
                secure_postgresql_database "$db_name" "$user" "$perms"
                ;;
            *)
                echo "Unsupported database engine: $engine"
                continue
                ;;
        esac
        
        # Setup automated backups for this database
        setup_database_backup "$db_name" "$domain" "$engine"
    done < <(get_database_config)
}

# Function to secure MySQL databases
secure_mysql_database() {
    local db_name="$1"
    local user="$2"
    local perms="$3"
    
    mysql -e "
        -- Remove any excessive privileges
        REVOKE ALL PRIVILEGES ON *.* FROM '$user'@'localhost';
        
        -- Grant only necessary permissions based on CloudPanel settings
        GRANT $perms ON $db_name.* TO '$user'@'localhost';
        
        -- Ensure proper security settings
        ALTER USER '$user'@'localhost' 
            REQUIRE SSL 
            PASSWORD EXPIRE INTERVAL 90 DAY 
            FAILED_LOGIN_ATTEMPTS 3 
            PASSWORD_LOCK_TIME 2;
        
        -- Enable binary logging for this database
        SET GLOBAL binlog_do_db = CONCAT(@@binlog_do_db, ',$db_name');
        
        FLUSH PRIVILEGES;"
}

# Function to setup database backups
setup_database_backup() {
    local db_name="$1"
    local domain="$2"
    local engine="$3"
    
    # Create backup script for this database
    local backup_script="${BACKUP_DIR}/${domain}-backup.sh"
    
    cat > "$backup_script" <<EOF
#!/bin/bash

# Backup script for $domain database ($db_name)
TIMESTAMP=\$(date +%Y%m%d_%H%M%S)
BACKUP_PATH="${BACKUP_DIR}/${domain}/\${TIMESTAMP}"
mkdir -p "\$BACKUP_PATH"

# Perform backup based on engine type
case "$engine" in
    "mysql")
        mysqldump --single-transaction \\
                 --quick \\
                 --lock-tables=false \\
                 --routines \\
                 --triggers \\
                 --events \\
                 "$db_name" | gzip > "\$BACKUP_PATH/${db_name}.sql.gz"
        ;;
    "postgresql")
        PGPASSWORD="\$PGPASSWORD" pg_dump \\
            -h localhost \\
            -U "$user" \\
            --clean \\
            --if-exists \\
            "$db_name" | gzip > "\$BACKUP_PATH/${db_name}.sql.gz"
        ;;
esac

# Verify backup integrity
if gunzip -t "\$BACKUP_PATH/${db_name}.sql.gz"; then
    echo "Backup completed successfully for $db_name"
else
    echo "Backup verification failed for $db_name"
    exit 1
fi

# Implement backup rotation (keep last 7 days)
find "${BACKUP_DIR}/${domain}" -type f -mtime +7 -delete
EOF

    # Make backup script executable
    chmod +x "$backup_script"
    
    # Add to crontab if not already present
    if ! crontab -l | grep -q "$backup_script"; then
        (crontab -l 2>/dev/null; echo "0 2 * * * $backup_script") | crontab -
    fi
}

# Function to monitor database security
monitor_database_security() {
    echo "Setting up database security monitoring..."
    
    # Create monitoring queries based on engine type
    cat > "${LOG_DIR}/monitor-queries.sql" <<EOF
-- Check for users with excessive privileges
SELECT user, host, privilege_type 
FROM information_schema.user_privileges 
WHERE privilege_type IN ('SUPER', 'FILE', 'PROCESS');

-- Check for non-local connections
SELECT user, host, db, command, time 
FROM information_schema.processlist 
WHERE host NOT LIKE 'localhost%';

-- Check for failed login attempts
SELECT * FROM mysql.general_log 
WHERE argument LIKE '%Access denied%' 
AND event_time > NOW() - INTERVAL 1 HOUR;
EOF

    # Setup periodic monitoring
    cat > "/etc/cron.hourly/db-security-monitor" <<EOF
#!/bin/bash
mysql -N < "${LOG_DIR}/monitor-queries.sql" > "${LOG_DIR}/security-audit-\$(date +%Y%m%d_%H).log"
EOF

    chmod +x "/etc/cron.hourly/db-security-monitor"
}

# Main execution
main() {
    echo "Starting database security configuration..."
    
    # Create backup of current configurations
    backup_date=$(date +%Y%m%d_%H%M%S)
    mkdir -p "${CP_HOME}/backups/db_security_${backup_date}"
    
    # Implement security measures
    secure_databases
    
    # Setup monitoring
    monitor_database_security
    
    echo "Database security configuration completed successfully"
}

main "$@"

This script provides:

Database security based on CloudPanel's actual configuration
Automatic backup management for each database
Security monitoring and audit logging
Privilege management based on CloudPanel's database_user settings
Engine-specific security measures for MySQL/PostgreSQL

To implement this security system:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-db-security.sh

# Run the script
/usr/local/bin/cloudpanel-db-security.sh

Securing CloudPanel on Ubuntu 24.04 Part 7

Mon, 18 Nov 2024 00:00:00 GMT

Part 7: SSL Certificate Management and Security

Let's create a comprehensive SSL certificate management system:

# Create SSL certificate management script
sudo nano /usr/local/bin/cloudpanel-ssl-manager.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
SSL_DIR="${CP_HOME}/services/nginx/ssl-certificates"
LOG_DIR="${CP_HOME}/logs/ssl"

mkdir -p "$SSL_DIR" "$LOG_DIR"

# Function to get certificate information from CloudPanel database
get_certificate_info() {
    sqlite3 "$DB_PATH" "
        SELECT 
            c.id,
            c.site_id,
            c.uid,
            c.expires_at,
            c.type,
            s.domain_name,
            s.user
        FROM certificate c
        JOIN site s ON c.site_id = s.id
        WHERE c.default_certificate = 1;"
}

# Function to monitor certificate expiration
monitor_certificates() {
    echo "Monitoring SSL certificates..."
    local current_time=$(date +%s)
    
    while IFS='|' read -r cert_id site_id uid expires domain user; do
        # Convert expires_at to timestamp
        local expire_time=$(date -d "$expires" +%s)
        local days_remaining=$(( ($expire_time - $current_time) / 86400 ))
        
        echo "Checking certificate for $domain (Expires in $days_remaining days)"
        
        # Alert if certificate is expiring soon
        if [ $days_remaining -lt 30 ]; then
            handle_expiring_certificate "$domain" "$days_remaining" "$cert_id"
        fi
        
        # Verify certificate chain
        verify_certificate_chain "$domain" "$cert_id"
        
    done < <(get_certificate_info)
}

# Function to verify and secure certificates
verify_certificate_chain() {
    local domain="$1"
    local cert_id="$2"
    
    # Get certificate chain from database
    local cert_path="${SSL_DIR}/${domain}.crt"
    local key_path="${SSL_DIR}/${domain}.key"
    
    # Extract certificate data from database
    sqlite3 "$DB_PATH" "
        SELECT 
            certificate,
            private_key,
            certificate_chain
        FROM certificate 
        WHERE id = $cert_id;" | while IFS='|' read -r cert key chain; do
        
        # Save certificate files with proper permissions
        echo "$cert" > "$cert_path"
        echo "$key" > "$key_path"
        chmod 644 "$cert_path"
        chmod 600 "$key_path"
        
        # Verify certificate chain
        if ! openssl verify -CAfile <(echo "$chain") "$cert_path" > /dev/null 2>&1; then
            log_ssl_error "$domain" "Certificate chain verification failed"
        fi
        
        # Verify certificate matches private key
        cert_modulus=$(openssl x509 -noout -modulus -in "$cert_path" | openssl md5)
        key_modulus=$(openssl rsa -noout -modulus -in "$key_path" | openssl md5)
        
        if [ "$cert_modulus" != "$key_modulus" ]; then
            log_ssl_error "$domain" "Certificate and private key mismatch"
        fi
    done
}

# Function to handle expiring certificates
handle_expiring_certificate() {
    local domain="$1"
    local days="$2"
    local cert_id="$3"
    
    echo "Certificate for $domain is expiring in $days days"
    
    # Log the expiration warning
    local message="SSL Certificate Expiration Warning for $domain"
    local details="Certificate will expire in $days days"
    
    # Update CloudPanel notification system
    sqlite3 "$DB_PATH" "
        INSERT INTO notification (
            created_at, 
            updated_at, 
            severity, 
            subject, 
            message,
            is_read
        ) VALUES (
            datetime('now'),
            datetime('now'),
            'warning',
            '$message',
            '$details',
            0
        );"
}

# Function to update Nginx SSL configuration
update_nginx_ssl_config() {
    local domain="$1"
    local cert_id="$2"
    
    # Get site configuration path
    local nginx_conf="${CP_HOME}/services/nginx/sites-enabled/${domain}.conf"
    
    # Update SSL configuration
    cat > "$nginx_conf" <<EOF
server {
    listen 443 ssl http2;
    server_name ${domain};
    
    # SSL Configuration
    ssl_certificate ${SSL_DIR}/${domain}.crt;
    ssl_certificate_key ${SSL_DIR}/${domain}.key;
    
    # Modern SSL Configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    
    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    
    # Security Headers
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Content-Type-Options nosniff;
    
    # Additional security headers for SSL
    add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval'; frame-ancestors 'self'";
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name ${domain};
    return 301 https://$server_name$request_uri;
}
EOF
}

# Function to log SSL-related events
log_ssl_event() {
    local domain="$1"
    local message="$2"
    local level="$3"
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $domain: $message" >> "${LOG_DIR}/ssl-events.log"
}

# Main execution
main() {
    echo "Starting SSL certificate management..."
    
    # Create backup of current SSL configurations
    backup_date=$(date +%Y%m%d_%H%M%S)
    backup_dir="${CP_HOME}/backups/ssl_${backup_date}"
    mkdir -p "$backup_dir"
    cp -r "$SSL_DIR" "$backup_dir/"
    
    # Monitor and update certificates
    monitor_certificates
    
    # Set up daily certificate monitoring
    if ! crontab -l | grep -q "cloudpanel-ssl-manager.sh"; then
        (crontab -l 2>/dev/null; echo "0 0 * * * /usr/local/bin/cloudpanel-ssl-manager.sh") | crontab -
    fi
    
    echo "SSL certificate management completed successfully"
}

main "$@"

This script provides:

Comprehensive SSL certificate management based on CloudPanel's database structure
Automatic monitoring of certificate expiration
Certificate chain verification
Private key security
Nginx SSL configuration management
Integration with CloudPanel's notification system

To implement this SSL management system:

# Make the script executable
chmod +x /usr/local/bin/cloudpanel-ssl-manager.sh

# Run the script
/usr/local/bin/cloudpanel-ssl-manager.sh

Securing CloudPanel on Ubuntu 24.04 Part 8

Tue, 19 Nov 2024 00:00:00 GMT

Part 8: User Access Management and Security Hardening

Let's create a comprehensive user security management system that integrates with CloudPanel.

# Create user security management script
sudo nano /usr/local/bin/cloudpanel-user-security.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
LOG_DIR="${CP_HOME}/logs/security"

mkdir -p "$LOG_DIR"

# Function to get user information from CloudPanel database
get_user_info() {
    sqlite3 "$DB_PATH" "
        SELECT 
            u.id,
            u.user_name,
            u.role,
            u.email,
            u.mfa,
            u.status,
            GROUP_CONCAT(s.domain_name) as sites
        FROM user u
        LEFT JOIN user_sites us ON u.id = us.user_id
        LEFT JOIN site s ON us.site_id = s.id
        GROUP BY u.id;"
}

# Function to get SSH user configurations
get_ssh_users() {
    sqlite3 "$DB_PATH" "
        SELECT 
            ss.user_name,
            ss.ssh_keys,
            s.domain_name,
            s.user
        FROM ssh_user ss
        JOIN site s ON ss.site_id = s.id;"
}

# Function to enforce user security policies
enforce_user_security() {
    echo "Enforcing user security policies..."
    
    while IFS='|' read -r user_id username role email mfa status sites; do
        echo "Processing security for user: $username"
        
        # Check and enforce MFA status
        if [ "$mfa" -eq 0 ]; then
            log_security_warning "$username" "MFA not enabled"
            enforce_mfa_requirement "$user_id" "$username"
        fi
        
        # Configure user access restrictions
        configure_user_access "$username" "$role" "$sites"
        
        # Set up user activity monitoring
        setup_user_monitoring "$username" "$role"
    done < <(get_user_info)
}

# Function to enforce MFA requirement
enforce_mfa_requirement() {
    local user_id="$1"
    local username="$2"
    
    echo "Enforcing MFA for user: $username"
    
    # Log MFA requirement
    sqlite3 "$DB_PATH" "
        INSERT INTO notification (
            created_at,
            updated_at,
            severity,
            subject,
            message,
            is_read
        ) VALUES (
            datetime('now'),
            datetime('now'),
            'warning',
            'MFA Required',
            'MFA enablement required for user: $username',
            0
        );"
}

# Function to configure SSH access
configure_ssh_access() {
    echo "Configuring SSH access..."
    
    while IFS='|' read -r ssh_user ssh_keys domain site_user; do
        local user_home="/home/$site_user"
        local ssh_dir="$user_home/.ssh"
        
        echo "Setting up SSH access for $ssh_user on $domain"
        
        # Create SSH directory with proper permissions
        mkdir -p "$ssh_dir"
        chmod 700 "$ssh_dir"
        
        # Configure authorized keys
        echo "$ssh_keys" > "$ssh_dir/authorized_keys"
        chmod 600 "$ssh_dir/authorized_keys"
        chown -R "$site_user:$site_user" "$ssh_dir"
        
        # Configure SSH restrictions
        cat >> "/etc/ssh/sshd_config.d/$ssh_user.conf" <<EOF
Match User $ssh_user
    ChrootDirectory $user_home
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
    PasswordAuthentication no
EOF
    done < <(get_ssh_users)
    
    # Restart SSH service
    systemctl restart sshd
}

# Function to configure user access controls
configure_user_access() {
    local username="$1"
    local role="$2"
    local sites="$3"
    
    echo "Configuring access controls for $username"
    
    # Set up role-based access restrictions
    case "$role" in
        "admin")
            setup_admin_access "$username"
            ;;
        "user")
            setup_user_access "$username" "$sites"
            ;;
        *)
            echo "Unknown role: $role"
            return 1
            ;;
    esac
}

# Function to set up admin access
setup_admin_access() {
    local username="$1"
    
    # Configure sudo access for admin users
    echo "$username ALL=(ALL:ALL) NOPASSWD: /usr/local/bin/cloudpanel-*" > "/etc/sudoers.d/$username"
    chmod 440 "/etc/sudoers.d/$username"
}

# Function to set up regular user access
setup_user_access() {
    local username="$1"
    local sites="$2"
    
    # Create restricted access configuration
    for site in ${sites//,/ }; do
        local site_path="/home/clp/htdocs/$site"
        
        # Set up ACLs for site access
        setfacl -R -m "u:$username:rx" "$site_path"
        setfacl -R -m "d:u:$username:rx" "$site_path"
    done
}

# Function to setup user activity monitoring
setup_user_monitoring() {
    local username="$1"
    local role="$2"
    
    # Configure audit logging for user actions
    auditctl -w "/home/$username" -p warx -k user_files
    
    # Set up process monitoring
    cat > "/etc/systemd/system/user-monitor-$username.service" <<EOF
[Unit]
Description=User Activity Monitor for $username
After=network.target

[Service]
ExecStart=/usr/bin/auditd -f -n
StandardOutput=append:${LOG_DIR}/user-${username}.log
StandardError=append:${LOG_DIR}/user-${username}-error.log

[Install]
WantedBy=multi-user.target
EOF

    systemctl daemon-reload
    systemctl enable "user-monitor-$username"
    systemctl start "user-monitor-$username"
}

# Function to log security events
log_security_event() {
    local username="$1"
    local message="$2"
    local level="$3"
    
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $username: $message" >> "${LOG_DIR}/security-events.log"
    
    # Add to CloudPanel event log
    sqlite3 "$DB_PATH" "
        INSERT INTO event (
            created_at,
            user_name,
            user_role,
            event_name,
            event_data
        ) VALUES (
            datetime('now'),
            '$username',
            '$role',
            'security_event',
            '$message'
        );"
}

# Main execution
main() {
    echo "Starting user security management..."
    
    # Create backup of current configurations
    backup_date=$(date +%Y%m%d_%H%M%S)
    backup_dir="${CP_HOME}/backups/user_security_${backup_date}"
    mkdir -p "$backup_dir"
    
    # Enforce security policies
    enforce_user_security
    
    # Configure SSH access
    configure_ssh_access
    
    echo "User security management completed successfully"
}

main "$@"

This script provides:

Comprehensive user security management based on CloudPanel's database structure
MFA enforcement and monitoring
Role-based access control implementation
SSH key management and restrictions
User activity monitoring and logging
Integration with CloudPanel's event and notification systems

Securing CloudPanel on Ubuntu 24.04 Part 9

Wed, 20 Nov 2024 00:00:00 GMT

Part 9: Comprehensive Security Auditing and Performance Monitoring

Let's create an integrated monitoring and auditing system:

# Create comprehensive monitoring script
sudo nano /usr/local/bin/cloudpanel-security-monitor.sh

#!/bin/bash

# Initialize paths and configuration
CP_HOME="/home/clp"
DB_PATH="${CP_HOME}/htdocs/app/data/db.sq3"
LOG_DIR="${CP_HOME}/logs/monitoring"
AUDIT_DIR="${CP_HOME}/logs/audit"

# Create necessary directories
mkdir -p "$LOG_DIR" "$AUDIT_DIR"

# Function to record system metrics in CloudPanel's database
record_system_metrics() {
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    
    # Record CPU usage
    local cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}')
    sqlite3 "$DB_PATH" "
        INSERT INTO instance_cpu (created_at, value)
        VALUES ('$timestamp', $cpu_usage);"

    # Record memory usage
    local memory_usage=$(free | grep Mem | awk '{print ($3/$2 * 100)}')
    sqlite3 "$DB_PATH" "
        INSERT INTO instance_memory (created_at, value)
        VALUES ('$timestamp', $memory_usage);"

    # Record load averages
    local load_1min=$(uptime | awk '{print $(NF-2)}' | tr -d ',')
    local load_5min=$(uptime | awk '{print $(NF-1)}' | tr -d ',')
    local load_15min=$(uptime | awk '{print $NF}')
    
    sqlite3 "$DB_PATH" "
        INSERT INTO instance_load_average (created_at, period, value)
        VALUES 
            ('$timestamp', 1, $load_1min),
            ('$timestamp', 5, $load_5min),
            ('$timestamp', 15, $load_15min);"

    # Record disk usage for each mount point
    df -P | grep '^/' | while read filesystem size used avail use mountpoint; do
        sqlite3 "$DB_PATH" "
            INSERT INTO instance_disk_usage (created_at, disk, value)
            VALUES ('$timestamp', '$mountpoint', ${use%\%});"
    done
}

# Function to analyze security events from CloudPanel's event table
analyze_security_events() {
    echo "Analyzing security events..."
    
    # Get recent security events
    sqlite3 "$DB_PATH" "
        SELECT 
            created_at,
            user_name,
            user_role,
            event_name,
            event_data,
            source_ip_address,
            user_agent
        FROM event
        WHERE created_at > datetime('now', '-1 hour')
        ORDER BY created_at DESC;" | while IFS='|' read -r timestamp username role event_name data ip agent; do
        
        # Analyze for suspicious patterns
        case "$event_name" in
            "login_failed")
                check_brute_force_attempts "$username" "$ip"
                ;;
            "file_modified")
                verify_file_changes "$data"
                ;;
            "config_changed")
                audit_configuration_change "$username" "$data"
                ;;
        esac
    done
}

# Function to check for brute force attempts
check_brute_force_attempts() {
    local username="$1"
    local ip="$2"
    
    # Count recent failed login attempts
    local failed_attempts=$(sqlite3 "$DB_PATH" "
        SELECT COUNT(*) FROM event 
        WHERE event_name = 'login_failed'
        AND source_ip_address = '$ip'
        AND created_at > datetime('now', '-1 hour');")
    
    if [ "$failed_attempts" -gt 5 ]; then
        # Log the security incident
        log_security_incident "Possible brute force attack" \
            "Multiple failed login attempts from IP: $ip targeting user: $username"
        
        # Block the IP address through CloudPanel's firewall
        sqlite3 "$DB_PATH" "
            INSERT INTO blocked_ip (
                site_id,
                created_at,
                updated_at,
                ip
            ) VALUES (
                NULL,
                datetime('now'),
                datetime('now'),
                '$ip'
            );"
    fi
}

# Function to verify file changes
verify_file_changes() {
    local file_data="$1"
    
    # Extract file path from event data
    local file_path=$(echo "$file_data" | jq -r '.path')
    
    if [ -f "$file_path" ]; then
        # Calculate current file hash
        local current_hash=$(sha256sum "$file_path" | cut -d' ' -f1)
        
        # Compare with expected hash if available
        if [ -f "${file_path}.hash" ]; then
            local expected_hash=$(cat "${file_path}.hash")
            if [ "$current_hash" != "$expected_hash" ]; then
                log_security_incident "Unexpected file modification" \
                    "File $file_path has been modified unexpectedly"
            fi
        else
            # Store initial hash for future comparison
            echo "$current_hash" > "${file_path}.hash"
        fi
    fi
}

# Function to audit configuration changes
audit_configuration_change() {
    local username="$1"
    local config_data="$2"
    
    # Log the configuration change
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] User $username modified configuration: $config_data" >> "$AUDIT_DIR/config_changes.log"
    
    # Store in CloudPanel's event system
    sqlite3 "$DB_PATH" "
        INSERT INTO event (
            created_at,
            user_name,
            user_role,
            event_name,
            event_data
        ) VALUES (
            datetime('now'),
            '$username',
            (SELECT role FROM user WHERE user_name = '$username'),
            'config_audit',
            '$config_data'
        );"
}

# Function to generate security reports
generate_security_report() {
    local report_file="$AUDIT_DIR/security_report_$(date +%Y%m%d).txt"
    
    echo "CloudPanel Security Audit Report - $(date)" > "$report_file"
    echo "==========================================" >> "$report_file"
    
    # System Metrics Summary
    echo "System Performance Metrics:" >> "$report_file"
    sqlite3 "$DB_PATH" "
        SELECT 
            avg(value) as avg_cpu,
            max(value) as max_cpu
        FROM instance_cpu 
        WHERE created_at > datetime('now', '-24 hours');" >> "$report_file"
    
    # Security Events Summary
    echo -e "\nSecurity Events Summary:" >> "$report_file"
    sqlite3 "$DB_PATH" "
        SELECT 
            event_name,
            COUNT(*) as event_count
        FROM event
        WHERE created_at > datetime('now', '-24 hours')
        GROUP BY event_name;" >> "$report_file"
    
    # Failed Login Attempts
    echo -e "\nFailed Login Attempts:" >> "$report_file"
    sqlite3 "$DB_PATH" "
        SELECT 
            source_ip_address,
            COUNT(*) as attempt_count
        FROM event
        WHERE event_name = 'login_failed'
        AND created_at > datetime('now', '-24 hours')
        GROUP BY source_ip_address
        HAVING attempt_count > 3;" >> "$report_file"
}

# Main execution loop
main() {
    echo "Starting security monitoring system..."
    
    while true; do
        # Record system metrics
        record_system_metrics
        
        # Analyze security events
        analyze_security_events
        
        # Generate daily report at midnight
        if [ "$(date +%H:%M)" = "00:00" ]; then
            generate_security_report
        fi
        
        # Sleep for monitoring interval
        sleep 300
    done
}

main "$@"

This monitoring system provides:

Comprehensive system metrics tracking using CloudPanel's database structure
Security event analysis and correlation
Automated threat detection and response
Configuration change auditing
Daily security reports
Integration with CloudPanel's event logging system

CloudPanel Sync Script Production to Staging

Wed, 08 Jan 2025 00:00:00 GMT

Why Automate CloudPanel Synchronization?

Managing multiple WordPress environments is a crucial part of modern web development. Whether you're testing updates, developing new features, or troubleshooting issues, having an automated synchronization system between your production and staging environments is essential. This guide will walk you through creating and optimizing a robust synchronization script for CloudPanel environments.

Understanding the Synchronization Process

Before diving into the code, let's understand what our script accomplishes:

Safely exports the production database
Transfers website files while preserving critical configurations
Updates database URLs and settings for the staging environment
Maintains proper permissions and security
Handles cache management

The Synchronization Script

Let's look at the synchronization script with error handling, logging, and security features:

#!/bin/bash
# =========================================================================== #
# Description:       CloudPanel Sync Script
# Purpose:           Synchronize WordPress sites between production and staging
# Environment:       CloudPanel (Debian/Ubuntu)
# Requirements:      CloudPanel, SSH key authentication
# =========================================================================== #

# Script configuration
set -euo pipefail  # Enhanced error handling
trap 'echo "Error on line $LINENO"' ERR

# Configuration variables
declare -A config=(
    ["PROD_DOMAIN"]="example.com"
    ["PROD_USER"]="prod-user"
    ["STAGING_DOMAIN"]="staging.example.com"
    ["STAGING_USER"]="staging-user"
    ["REMOTE_SSH"]="root@production-ip"
    ["TABLE_PREFIX"]="wp_"
)

# Path configurations
declare -A paths=(
    ["PROD_PATH"]="/home/${config[PROD_USER]}/htdocs/${config[PROD_DOMAIN]}"
    ["STAGING_PATH"]="/home/${config[STAGING_USER]}/htdocs/${config[STAGING_DOMAIN]}"
    ["LOG_FILE"]="/home/${config[STAGING_USER]}/sync.log"
)

# Enhanced logging function
log() {
    local level=$1
    shift
    echo "[$(date -u '+%Y-%m-%d %H:%M:%S UTC')] [$level] $*" | tee -a "${paths[LOG_FILE]}"
}

# Validation functions
validate_paths() {
    if [[ ! -d "${paths[STAGING_PATH]}" ]]; then
        log "ERROR" "Staging directory ${paths[STAGING_PATH]} not found"
        exit 1
    fi
}

# Database backup function with compression
backup_database() {
    log "INFO" "Starting database backup"
    ssh "${config[REMOTE_SSH]}" "clpctl db:export --databaseName=${config[PROD_USER]} \
        --file=/home/${config[PROD_USER]}/tmp/backup.sql.gz"
}

# Enhanced database sync with progress
sync_database() {
    log "INFO" "Synchronizing database"
    rsync -azP --info=progress2 \
        "${config[REMOTE_SSH]}:/home/${config[PROD_USER]}/tmp/backup.sql.gz" \
        "/home/${config[STAGING_USER]}/tmp/"
}

# Main execution flow
main() {
    log "INFO" "Starting synchronization process"
    
    # Initial validation
    validate_paths
    
    # Backup and sync
    backup_database
    sync_database
    
    # Update database URLs
    log "INFO" "Updating database URLs"
    mysql --defaults-extra-file="/home/${config[STAGING_USER]}/my.cnf" \
        -D "${config[STAGING_USER]}" -e "
        UPDATE ${config[TABLE_PREFIX]}options 
        SET option_value = REPLACE(option_value, 
            'https://${config[PROD_DOMAIN]}', 
            'https://${config[STAGING_DOMAIN]}')
        WHERE option_name IN ('home', 'siteurl');"
    
    # File synchronization with exclusions
    log "INFO" "Syncing files"
    rsync -azP --delete \
        --exclude={'wp-config.php','.user.ini','wp-content/cache/*'} \
        "${config[REMOTE_SSH]}:${paths[PROD_PATH]}/" \
        "${paths[STAGING_PATH]}/"
    
    # Security measures
    log "INFO" "Applying security settings"
    mysql --defaults-extra-file="/home/${config[STAGING_USER]}/my.cnf" \
        -D "${config[STAGING_USER]}" \
        -e "UPDATE ${config[TABLE_PREFIX]}options 
            SET option_value = '0' 
            WHERE option_name = 'blog_public';"
    
    # Cleanup
    log "INFO" "Performing cleanup"
    rm -f "/home/${config[STAGING_USER]}/tmp/backup.sql.gz"
    
    log "SUCCESS" "Synchronization completed"
}

# Execute main function
main

Key implementation and Best Practices

Let's break down the key implementation in this enhanced version:

1. Structured Configuration

We've organized configuration variables into associative arrays, making the script more maintainable and easier to update:

declare -A config=(
    ["PROD_DOMAIN"]="example.com"
    ["STAGING_DOMAIN"]="staging.example.com"
    # ... other configurations
)

2. Enhanced Error Handling

The script now includes proper error trapping and handling:

set -euo pipefail
trap 'echo "Error on line $LINENO"' ERR

3. Robust Logging

A dedicated logging function provides consistent and detailed logging:

log() {
    local level=$1
    shift
    echo "[$(date -u '+%Y-%m-%d %H:%M:%S UTC')] [$level] $*" | tee -a "${paths[LOG_FILE]}"
}

Common Scenarios and Solutions

Scenario 1: Large File Transfers

When dealing with large WordPress installations:

# Add these rsync options for large transfers
rsync -azP --partial --partial-dir=.rsync-partial \
    --exclude={'wp-config.php','.user.ini','wp-content/cache/*'} \
    "${config[REMOTE_SSH]}:${paths[PROD_PATH]}/" \
    "${paths[STAGING_PATH]}/"

Scenario 2: Custom Plugin Configurations

For sites with specific plugin configurations:

# Add to the database update queries
mysql --defaults-extra-file="/home/${config[STAGING_USER]}/my.cnf" \
    -D "${config[STAGING_USER]}" -e "
    UPDATE ${config[TABLE_PREFIX]}options 
    SET option_value = REPLACE(
        option_value, 
        'production-specific-setting', 
        'staging-specific-setting'
    )
    WHERE option_name LIKE '%plugin_setting%';"

Scenario 3: Handling Multiple Sites

For managing multiple WordPress installations:

# Create a sites configuration file
cat > sites.conf << EOF
site1:example1.com:user1
site2:example2.com:user2
EOF

# Modified script to handle multiple sites
while IFS=: read -r site domain user; do
    config["PROD_DOMAIN"]=$domain
    config["PROD_USER"]=$user
    main
done < sites.conf

Best Practices for Production Use

Always Test First: Run the script on a test environment before implementing in production.

Backup Strategy: Implement proper backup procedures:

# Add to the script before main sync
backup_date=$(date +%Y%m%d_%H%M%S)
clpctl backup:create --name="pre_sync_${backup_date}"

Security Considerations: Use SSH keys and restrict permissions:

chmod 600 /home/${config[STAGING_USER]}/my.cnf
chmod 700 /home/${config[STAGING_USER]}/tmp

Monitoring and Maintenance

To ensure your synchronization system remains reliable:

Log Rotation: Implement log rotation to manage log files:

# Add to crontab
0 0 * * * logrotate -f /etc/logrotate.d/cloudpanel-sync

Health Checks: Add post-sync validation:

# Add to the main function
check_wordpress_health() {
    curl -sL "https://${config[STAGING_DOMAIN]}/wp-admin" | grep -q "wp-login"
    return $?
}

Conclusion

This synchronization script provides a robust foundation for managing WordPress environments in CloudPanel. By implementing and following the best practices, you can ensure reliable and efficient synchronization between your production and staging environments.

Remember to always test the script in a controlled environment first and maintain proper backups before running any synchronization operations. The script can be further customized based on your specific needs and requirements.

For more advanced usage and customization options, open a discussion on our forum https://forum.hhf.technology/.

Guide-Building & Deploying Naxsi WAF on CloudPanel

Wed, 08 Jan 2025 00:00:00 GMT

Comprehensive Guide: Building and Deploying Naxsi WAF on CloudPanel

⚠️ CRITICAL ENVIRONMENT NOTICE

Execution Environment Requirements

WARNING: This operation must be executed exclusively in a non-production environment.

Risk Assessment:

System stability may be affected during module compilation and installation
Web services will require restart
Existing NGINX configuration may need modifications
No automatic rollback mechanisms are available

Required Environment:

Development/staging CloudPanel instance
Complete system backup
Isolated network segment
Non-production workload

DO NOT PROCEED with execution in any production environment under any circumstances.

Understanding CloudPanel's NGINX Structure

CloudPanel uses a specific NGINX configuration structure that we need to work with:

Main configuration: /etc/nginx/nginx.conf
Site configurations: /etc/nginx/sites-enabled/
Module configurations: /etc/nginx/modules-available/ and /etc/nginx/modules-enabled/
CloudPanel vhost template: Located in CloudPanel's configuration directory

Prerequisites

First, switch to root user and install the required dependencies:

sudo su

# Install build essentials and required libraries
apt update
apt install -y build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev wget git

# Install additional CloudPanel-specific dependencies
apt install -y libmodsecurity3 libmodsecurity-dev

Download and Prepare Build Environment

Create a dedicated build directory and download required components:

# Create build directory
mkdir -p /root/naxsi-build
cd /root/naxsi-build

# Get NGINX version from current installation
NGINX_VERSION=$(nginx -v 2>&1 | grep -o '[0-9.]*$')
echo "Building for NGINX version: $NGINX_VERSION"

# Download NGINX source
wget "https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz"
tar -xzf nginx-${NGINX_VERSION}.tar.gz

# Download Naxsi
NAXSI_VERSION=1.6
wget "https://github.com/wargio/naxsi/releases/download/${NAXSI_VERSION}/naxsi-${NAXSI_VERSION}-src-with-deps.tar.gz"
tar -xzf naxsi-${NAXSI_VERSION}-src-with-deps.tar.gz

Compile Naxsi Module

cd /root/naxsi-build/nginx-${NGINX_VERSION}

# Get existing NGINX configuration
NGINX_CONFIGURE_ARGS=$(nginx -V 2>&1 | grep "configure arguments:" | cut -d " " -f2-)

# Configure NGINX with Naxsi
./configure $NGINX_CONFIGURE_ARGS --add-dynamic-module=../naxsi/naxsi_src/

# Compile only the modules
make modules

Install Naxsi Module

# Create modules directory if it doesn't exist
mkdir -p /usr/lib/nginx/modules

# Copy the compiled module
cp objs/ngx_http_naxsi_module.so /usr/lib/nginx/modules/

# Copy Naxsi rules
mkdir -p /etc/nginx/naxsi
cp ../naxsi/naxsi_rules/* /etc/nginx/naxsi/

Configure Naxsi in CloudPanel

Create module configuration:

# Create Naxsi module configuration
cat > /etc/nginx/modules-available/mod-naxsi.conf << 'EOL'
load_module modules/ngx_http_naxsi_module.so;
EOL

# Enable the module
ln -s /etc/nginx/modules-available/mod-naxsi.conf /etc/nginx/modules-enabled/50-mod-naxsi.conf

Create base Naxsi configuration:

# Create base configuration
cat > /etc/nginx/naxsi/naxsi-base.conf << 'EOL'
# Basic configuration
SecRulesEnabled;
LibInjectionSql;
LibInjectionXss;
DeniedUrl "/RequestDenied";

# Core rules
include /etc/nginx/naxsi/naxsi_core.rules;

# Check rules
CheckRule "$SQL >= 8" BLOCK;
CheckRule "$XSS >= 8" BLOCK;
CheckRule "$RFI >= 8" BLOCK;
CheckRule "$UPLOAD >= 8" BLOCK;
CheckRule "$LIBINJECTION_SQL >= 8" BLOCK;
CheckRule "$LIBINJECTION_XSS >= 8" BLOCK;
EOL

Integrating with CloudPanel Vhost Configuration

Based on the provided CloudPanel configuration template, we'll add Naxsi support while maintaining CloudPanel's functionality. Create a Naxsi configuration snippet:

cat > /etc/nginx/snippets/naxsi-location.conf << 'EOL'
# Naxsi protection
include /etc/nginx/naxsi/naxsi-base.conf;

# Common WordPress whitelist rules
BasicRule wl:1315 "mz:$BODY_VAR:comment";
BasicRule wl:1315 "mz:$BODY_VAR:content";
BasicRule wl:1315 "mz:$BODY_VAR:excerpt";
BasicRule wl:1315 "mz:$BODY_VAR:title";
BasicRule wl:1315 "mz:$BODY_VAR:post_content";

# Add request denied location
location /RequestDenied {
    internal;
    return 403;
}
EOL

Modify Site Configuration

For each site you want to protect with Naxsi, modify the configuration to include Naxsi rules. Here's an example based on the provided CloudPanel configuration:

server {
    listen 80;
    listen [::]:80;
    server_name example.com;

    # Include Naxsi base configuration
    include /etc/nginx/naxsi/naxsi-base.conf;

    location / {
        # Include Naxsi location rules
        include /etc/nginx/snippets/naxsi-location.conf;

        # Existing CloudPanel proxy configuration
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://backend;
    }

    # RequestDenied location (required by Naxsi)
    location /RequestDenied {
        internal;
        return 403;
    }
}

Testing and Verification

Verify NGINX configuration:

nginx -t

Test Naxsi module loading:

nginx -V 2>&1 | grep naxsi

Restart NGINX:

systemctl restart nginx

Test Naxsi protection:

# Test SQL injection protection
curl "http://your-site.com/?id=1%20OR%201=1"

# Test XSS protection
curl "http://your-site.com/?<script>alert(1)</script>"

Common Issues and Troubleshooting

Module Loading Issues:
- Verify module path in configuration
- Check NGINX error logs: tail -f /var/log/nginx/error.log
- Ensure proper file permissions on module and configuration files
Rule Conflicts:
- Start with learning mode: Add LearningMode; to your configuration
- Monitor logs for false positives
- Create specific whitelists as needed
CloudPanel Compatibility:
- Keep CloudPanel's proxy settings intact
- Maintain proper order of includes and configurations
- Test thoroughly after any configuration changes

Maintenance and Updates

Regular tasks:
- Monitor NGINX and Naxsi logs
- Update rules based on false positives/negatives
- Keep track of Naxsi updates and security patches
Backup procedures:
- Maintain copies of custom rules and configurations
- Document all modifications to CloudPanel configurations
- Keep build environment for future updates

Security Best Practices

Rule Management:
- Start with strict rules and whitelist as needed
- Document all rule modifications
- Regular review of whitelisted rules
Monitoring:
- Enable detailed logging for blocked requests
- Set up alerts for repeated attacks
- Regular review of Naxsi logs
Performance:
- Monitor system resources after enabling Naxsi
- Optimize rules for your specific use case
- Regular cleanup of log files

Remember to always test thoroughly in a staging environment before deploying to production. Keep detailed documentation of all modifications and custom rules for future reference.

Practical Naxsi WAF Implementation Examples for CloudPanel

Wed, 08 Jan 2025 00:00:00 GMT

Practical Naxsi WAF Implementation Examples for CloudPanel

This document provides detailed examples and practical configurations for implementing Naxsi WAF in CloudPanel environments. Each example includes explanations of the rules and their purposes, helping you understand how to protect different types of web applications.

Basic WordPress Protection

WordPress sites in CloudPanel often need special consideration due to their dynamic nature. Here's a comprehensive configuration that balances security with functionality:

# Include in the server block of your WordPress site
server {
    # Standard CloudPanel configuration remains unchanged
    listen 80;
    listen [::]:80;
    server_name example.com;
    
    # Enable Naxsi base rules
    include /etc/nginx/naxsi/naxsi-base.conf;
    
    # WordPress-specific whitelist rules
    include /etc/nginx/naxsi/wordpress.rules;
    
    location / {
        # Standard WordPress handling
        try_files $uri $uri/ /index.php?$args;
        
        # WordPress-specific Naxsi configurations
        BasicRule wl:1315,1101 "mz:$BODY_VAR:post_title";
        BasicRule wl:1315,1101 "mz:$BODY_VAR:content";
        BasicRule wl:1315 "mz:$BODY_VAR:excerpt";
        BasicRule wl:1101 "mz:$BODY_VAR:cat";
        
        # Allow WordPress admin actions
        BasicRule wl:1000,1015 "mz:$URL:/wp-admin/|$BODY_VAR:action";
    }
    
    # Special handling for wp-admin
    location /wp-admin {
        # Allow file uploads in wp-admin
        BasicRule wl:1500 "mz:$URL:/wp-admin/|$BODY_VAR:attachment";
        BasicRule wl:1310,1311 "mz:$URL:/wp-admin/|BODY";
        
        # Standard CloudPanel proxy configuration
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

This configuration protects against common WordPress attack vectors while allowing legitimate administrative actions. The whitelist rules are carefully crafted to permit essential WordPress functionality while maintaining security.

E-commerce Protection Example

For e-commerce sites running on CloudPanel, we need to balance strict security with user experience. Here's an example configuration for an e-commerce platform:

# E-commerce specific Naxsi rules
server {
    listen 80;
    listen [::]:80;
    server_name shop.example.com;
    
    # Basic Naxsi configuration
    include /etc/nginx/naxsi/naxsi-base.conf;
    
    # E-commerce specific rules
    location / {
        # Allow product searches with special characters
        BasicRule wl:1001,1015 "mz:$ARGS_VAR:q";
        
        # Allow price ranges in filters
        BasicRule wl:1015 "mz:$ARGS_VAR:price_min|$ARGS_VAR:price_max";
        
        # Allow product sorting
        BasicRule wl:1015 "mz:$ARGS_VAR:sort_by";
        
        # Standard CloudPanel proxy settings
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
    
    # Shopping cart protection
    location /cart {
        # Allow quantity updates
        BasicRule wl:1101 "mz:$BODY_VAR:quantity";
        
        # Allow product variants
        BasicRule wl:1015 "mz:$BODY_VAR:variant_id";
        
        # Strict XSS protection
        CheckRule "$XSS >= 4" BLOCK;
        CheckRule "$LIBINJECTION_XSS >= 4" BLOCK;
    }
    
    # Checkout process protection
    location /checkout {
        # Allow address input with special characters
        BasicRule wl:1315 "mz:$BODY_VAR:address";
        BasicRule wl:1315 "mz:$BODY_VAR:address2";
        
        # Allow postal codes with special formats
        BasicRule wl:1015 "mz:$BODY_VAR:postcode";
        
        # Extra protection for payment data
        CheckRule "$SQL >= 4" BLOCK;
        CheckRule "$RFI >= 2" BLOCK;
    }
}

API Protection Example

When protecting APIs in CloudPanel, we need specific configurations to handle JSON data and API-specific threats:

# API-specific Naxsi configuration
server {
    listen 80;
    listen [::]:80;
    server_name api.example.com;
    
    # Enable JSON parsing
    include /etc/nginx/naxsi/naxsi-base.conf;
    
    # API request handling
    location /api/v1 {
        # Allow JSON content type
        BasicRule wl:1200,1205 "mz:$HEADERS_VAR:content-type";
        
        # Allow complex JSON structures
        BasicRule wl:1310,1311 "mz:BODY";
        
        # Strict SQL injection protection
        CheckRule "$LIBINJECTION_SQL >= 4" BLOCK;
        CheckRule "$SQL >= 4" BLOCK;
        
        # Standard CloudPanel proxy configuration
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
    
    # Authentication endpoints
    location /api/v1/auth {
        # Allow JSON Web Tokens
        BasicRule wl:1100,1101 "mz:$BODY_VAR:token";
        
        # Allow email addresses
        BasicRule wl:1315 "mz:$BODY_VAR:email";
        
        # Extra protection for auth endpoints
        CheckRule "$XSS >= 2" BLOCK;
        CheckRule "$RFI >= 2" BLOCK;
    }
}

Learning Mode Configuration

During initial deployment, it's crucial to use learning mode to understand your application's behavior. Here's an example configuration:

# Learning mode configuration
server {
    listen 80;
    listen [::]:80;
    server_name staging.example.com;
    
    # Enable learning mode
    include /etc/nginx/naxsi/naxsi-base.conf;
    LearningMode;
    
    # Enable detailed logging
    set $naxsi_extensive_log 1;
    
    location / {
        # Log all potential violations
        BasicRule "log:all";
        
        # Standard CloudPanel proxy configuration
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
    
    # Custom log format for analysis
    log_format naxsi_learning '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for" '
                            'naxsi: "$naxsi_sig" "$naxsi_match"';
    
    access_log /var/log/nginx/naxsi_learning.log naxsi_learning;
}

Advanced Security Headers

Enhance your Naxsi protection with additional security headers:

# Security headers configuration
server {
    listen 80;
    listen [::]:80;
    server_name secure.example.com;
    
    # Include Naxsi base configuration
    include /etc/nginx/naxsi/naxsi-base.conf;
    
    # Add security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline';" always;
    
    location / {
        # Standard Naxsi protection
        include /etc/nginx/naxsi/naxsi-base.conf;
        
        # CloudPanel proxy configuration
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
}

File Upload Protection

For sites handling file uploads, configure specific protections:

# File upload protection
server {
    listen 80;
    listen [::]:80;
    server_name upload.example.com;
    
    # Include Naxsi base configuration
    include /etc/nginx/naxsi/naxsi-base.conf;
    
    # File upload location
    location /upload {
        # Allow file uploads but maintain security
        BasicRule wl:1500 "mz:$URL:/upload|BODY";
        
        # Limit file types
        if ($upload_filename ~* (?i).(php|phtml|php3|php4|php5|php6|php7|pht|phar|inc)$) {
            return 403;
        }
        
        # Additional upload protections
        client_max_body_size 10M;
        client_body_buffer_size 128k;
        
        # CloudPanel proxy configuration
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }
}

Common Naxsi Rules Explained

Understanding common Naxsi rules and their purposes:

# SQL Injection Protection
CheckRule "$SQL >= 8" BLOCK;          # Block SQL injection attempts
CheckRule "$LIBINJECTION_SQL >= 8" BLOCK;  # Use libinjection for SQL detection

# XSS Protection
CheckRule "$XSS >= 8" BLOCK;          # Block Cross-site scripting
CheckRule "$LIBINJECTION_XSS >= 8" BLOCK;  # Use libinjection for XSS detection

# Directory Traversal Protection
CheckRule "$TRAVERSAL >= 4" BLOCK;    # Block directory traversal attempts

# Remote File Inclusion Protection
CheckRule "$RFI >= 8" BLOCK;          # Block remote file inclusion

# File Upload Protection
CheckRule "$UPLOAD >= 8" BLOCK;       # Block malicious file uploads

Each rule has a score threshold that determines when to block requests. Lower scores increase security but may cause false positives.

Monitoring and Logging Configuration

Implement comprehensive logging for Naxsi:

# Logging configuration
http {
    # Define Naxsi log format
    log_format naxsi_json escape=json '{'
        '"timestamp":"$time_iso8601",'
        '"client":"$remote_addr",'
        '"method":"$request_method",'
        '"uri":"$uri",'
        '"args":"$args",'
        '"headers":{'
            '"user-agent":"$http_user_agent",'
            '"referer":"$http_referer"'
        '},'
        '"naxsi":{'
            '"match":"$naxsi_match",'
            '"sig":"$naxsi_sig",'
            '"score":"$naxsi_score",'
            '"matches":"$naxsi_matches"'
        '}'
    '}';
    
    # Enable Naxsi logging
    access_log /var/log/nginx/naxsi_access.log naxsi_json if=$naxsi_matched;
    error_log /var/log/nginx/naxsi_error.log;
}

These examples demonstrate common Naxsi configurations for different scenarios in CloudPanel. Remember to adapt these examples to your specific needs and always test thoroughly in a staging environment before deploying to production.

How to clean the commit history in git 🔍

Mon, 06 Jan 2025 00:00:00 GMT

To clean the commit history in Git, you can follow a series of steps that effectively remove all previous commits while keeping the current state of your code. This process is often used when the commit history contains unnecessary or sensitive information. Here’s how to do it:

Steps to Clean Git Commit History

Create an Orphan Branch: This step creates a new branch without any previous commit history.
```
git checkout --orphan new_branch
```
Add All Files: Stage all files in your working directory for the new commit.
```
git add -A
```
Commit the Changes: Make a new initial commit with all the files.
```
git commit -m "Initial commit"
```
Delete the Old Main Branch: Remove the old branch that contains the history. This action is irreversible.
```
git branch -D main
```
Rename the New Branch: Rename your newly created branch to main (or your preferred name).
```
git branch -m main
```
Force Push to Remote Repository: Update your remote repository to reflect these changes, effectively overwriting the history.
```
git push -f origin main
```

Important Considerations

Data Loss: This process permanently deletes all previous commits, so ensure you have backups if needed.
Collaboration Impact: If others are using the repository, communicate these changes as they will lose access to prior commit history.

This method is particularly useful for cleaning up repositories before public releases or when sensitive data has been committed unintentionally.

Mastering OpenCTI Deployment with Docker 🛡️

Wed, 04 Sep 2024 00:00:00 GMT

Understanding OpenCTI

OpenCTI (Open Cyber Threat Intelligence) represents a significant advancement in threat intelligence management. At its core, it's an open-source platform that transforms how organizations handle cyber threat intelligence, providing a comprehensive solution for collecting, analyzing, and sharing threat data.

Key Benefits

The platform offers several advantages that make it invaluable for security teams:

Centralized Intelligence Management: Consolidates threat data from multiple sources into a single, searchable platform
Advanced Analysis Capabilities: Enables deep analysis of threats through relationship mapping and visualization
Automated Data Processing: Reduces manual effort through automated ingestion and processing of threat intelligence
STIX/TAXII Compatible: Natively supports industry-standard threat intelligence formats
Extensible Architecture: Allows integration with various threat intelligence sources through connectors

System Architecture

OpenCTI's architecture is built on several key components, each serving a specific purpose:

Frontend: React-based web interface for user interaction
API: GraphQL API for data access and manipulation
Workers: Distributed processing units for handling tasks
Database Layer: Elasticsearch for data storage and search capabilities
Message Broker: RabbitMQ for reliable message queuing
Object Storage: MinIO for storing files and attachments
Cache Layer: Redis for performance optimization

Prerequisites

# Update system packages
apt update && apt upgrade -y

# Install required packages
apt install docker-compose jq -y

# Configure system settings
echo "vm.max_map_count=1048575" >> /etc/sysctl.conf
sysctl -p

# Create application directory
mkdir -p /opt/opencti && cd /opt/opencti

Deployment Configuration

Let's examine the deployment configuration in detail:

version: '3'
services:
  redis:
    image: redis:7.2.5
    restart: always
    volumes:
      - redisdata:/data

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.4
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536

  # Additional services configuration...

Environment Configuration

Create a .env file with necessary configurations:

cat << EOF > .env
OPENCTI_ADMIN_EMAIL=admin@opencti.io
OPENCTI_ADMIN_PASSWORD=ChangeMePlease
OPENCTI_ADMIN_TOKEN=$(cat /proc/sys/kernel/random/uuid)
OPENCTI_BASE_URL=http://localhost:8080
MINIO_ROOT_USER=$(cat /proc/sys/kernel/random/uuid)
MINIO_ROOT_PASSWORD=$(cat /proc/sys/kernel/random/uuid)
RABBITMQ_DEFAULT_USER=guest
RABBITMQ_DEFAULT_PASS=guest
ELASTIC_MEMORY_SIZE=8G
CONNECTOR_ANALYSIS_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_HISTORY_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_EXPORT_FILE_STIX_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_EXPORT_FILE_CSV_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_IMPORT_FILE_STIX_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_EXPORT_FILE_TXT_ID=$(cat /proc/sys/kernel/random/uuid)
CONNECTOR_IMPORT_DOCUMENT_ID=$(cat /proc/sys/kernel/random/uuid)
SMTP_HOSTNAME=$(hostname)
EOF

Integrating Threat Intelligence Sources

MITRE ATT&CK Integration

The MITRE connector provides comprehensive mapping of tactics, techniques, and procedures:

connector-mitre:
  image: opencti/connector-mitre:6.2.6
  environment:
    - OPENCTI_URL=http://opencti:8080
    - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
    - CONNECTOR_ID=95996acf-1b08-4492-b66f-f54a46f14398
    - "CONNECTOR_NAME=MITRE Datasets"
    - CONNECTOR_SCOPE=tool,report,malware,identity,campaign,intrusion-set,attack-pattern,course-of-action
    - MITRE_INTERVAL=7
  restart: always
  depends_on:
    - opencti

MalwareBazaar Integration

Configure the MalwareBazaar connector for malware intelligence:

connector-malwarebazaar:
  image: opencti/connector-malwarebazaar-recent-additions:6.2.6
  environment:
    - OPENCTI_URL=http://opencti:8080
    - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
    - CONNECTOR_ID=51ebc133-58c4-4a8b-9afc-4177e597cc91
    - MALWAREBAZAAR_RECENT_ADDITIONS_INCLUDE_TAGS=exe,dll,docm,docx,doc,xls,xlsx,xlsm,js
  restart: always
  depends_on:
    - opencti

Post-Deployment Steps

After deploying OpenCTI, several important steps should be taken:

Security Hardening:
- Change default passwords
- Configure SSL/TLS
- Set up network security groups
- Enable authentication for all services
Connector Verification:
- Check connector logs for proper initialization
- Verify data ingestion from each source
- Monitor connector performance
Performance Tuning:
- Adjust Elasticsearch settings based on data volume
- Configure worker counts for optimal processing
- Monitor and adjust cache settings
Backup Configuration:
- Set up regular backups of Elasticsearch data
- Configure MinIO backup strategy
- Document recovery procedures

Best Practices and Common Pitfalls

When operating OpenCTI, consider these important guidelines:

Resource Management:
- Monitor Elasticsearch heap usage
- Watch for worker queue buildup
- Regular cleanup of old data
Data Quality:
- Implement data validation procedures
- Regular auditing of imported data
- Maintain consistent tagging practices
Performance Optimization:
- Index optimization in Elasticsearch
- Proper caching configuration
- Regular maintenance windows

Troubleshooting Guide

Common issues and their solutions:

Elasticsearch Issues:

# Check cluster health
curl -X GET "localhost:9200/_cluster/health?pretty"

# Clear caches if needed
curl -X POST "localhost:9200/_cache/clear?pretty"

Connector Problems:

# View connector logs
docker logs -f opencti-connector-mitre

# Restart problematic connector
docker-compose restart connector-mitre

Conclusion

OpenCTI provides a robust platform for threat intelligence management, but proper deployment and maintenance are crucial for its effectiveness. Regular monitoring, updates, and optimization ensure the platform continues to serve its purpose in your security infrastructure.

Remember to regularly check for updates and new connector releases, as the threat intelligence landscape is constantly evolving.

Enterprise-Grade Infrastructure Monitoring with Zabbix 🔍

Thu, 02 Jan 2025 00:00:00 GMT

The Critical Role of Infrastructure Monitoring

In modern enterprise environments, infrastructure monitoring isn't just a best practice—it's a necessity. With distributed systems becoming increasingly complex, the ability to proactively monitor networks, systems, firewalls, and endpoints is crucial for maintaining high availability and security. Infrastructure monitoring provides the visibility needed to ensure 24/7 service reliability and rapid incident response.

Why Choose Zabbix?

Zabbix stands out in the monitoring landscape for several compelling reasons:

Open-source architecture with enterprise-grade capabilities
Unified monitoring platform for networks, systems, and IoT devices
Highly scalable architecture supporting thousands of nodes
Rich visualization and alerting capabilities
Active community and extensive documentation
No licensing costs for core functionality

Infrastructure Requirements

Before beginning the installation, ensure your environment meets these specifications:

System Prerequisites

Operating System: Ubuntu (LTS recommended)
CPU: 2 vCPUs minimum (4+ recommended for production)
Memory: 8GB RAM minimum (16GB+ for production)
Storage: 20GB minimum (plan for growth based on retention policies)
Network: Static IP and full root access

Component Architecture

Database: MySQL/MariaDB for metric storage
Web Stack: PHP, Nginx for frontend services
Network: Port 80/443 (Web UI), 10051 (Zabbix Server)

#!/bin/bash
# Add Zabbix repository and update system
sudo wget https://repo.zabbix.com/zabbix/7.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_7.0-2+ubuntu24.04_all.deb
sudo dpkg -i zabbix-release_7.0-2+ubuntu24.04_all.deb
sudo apt update -y

Core Installation Process

1. Database Configuration

First, we'll set up the MySQL database that will store all monitoring data:

CREATE DATABASE zabbix CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
CREATE USER zabbix@localhost IDENTIFIED BY 'mySecurePassword';
GRANT ALL PRIVILEGES ON zabbix.* TO zabbix@localhost;
SET GLOBAL log_bin_trust_function_creators = 1;

2. Component Installation

Install the required Zabbix components:

#!/bin/bash
# Install core components
sudo apt install -y zabbix-server-mysql \
                   zabbix-frontend-php \
                   zabbix-nginx-conf \
                   zabbix-sql-scripts \
                   zabbix-agent

# Import initial schema
zcat /usr/share/zabbix-sql-scripts/mysql/server.sql.gz | \
     mysql --default-character-set=utf8mb4 -uzabbix -p zabbix

# Reset database flag after import
mysql -uroot -p -e "SET GLOBAL log_bin_trust_function_creators = 0;"

3. Server Configuration

Configure the Zabbix server by editing the configuration file:

# Database configuration
DBHost=localhost
DBName=zabbix
DBUser=zabbix
DBPassword=mySecurePassword

# Performance tuning
StartPollers=5
StartPollersUnreachable=1
StartPingers=1
StartDiscoverers=1
StartHTTPPollers=1

# Cache settings
CacheSize=32M
HistoryCacheSize=16M
TrendCacheSize=4M

4. Web Interface Setup

Configure Nginx for the Zabbix frontend:

server {
    listen 8080;
    server_name zabbix.yourdomain.com;

    root /usr/share/zabbix;

    location / {
        index index.php;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.3-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
}

Advanced Configuration Tips

Performance Optimization

Database Tuning Optimize MySQL for better performance with Zabbix:

[mysqld]
innodb_buffer_pool_size = 4G
innodb_log_file_size = 1G
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 2
innodb_doublewrite = 0

Housekeeping Settings Configure data retention to manage database growth:

# Retention settings
HousekeepingFrequency=1
MaxHousekeeperDelete=5000

# History storage periods
HistoryStorageDateIndex=1
HistoryStorageGlobalIndex=1

Security Considerations

Network Security
- Implement strict firewall rules
- Use TLS for agent communication
- Consider implementing a reverse proxy
Authentication
- Enable LDAP/Active Directory integration
- Implement 2FA for administrative access
- Regular password rotation

Production Best Practices

High Availability Setup
- Configure active-passive server setup
- Implement database replication
- Use load balancers for web interface
Backup Strategy
- Regular database backups
- Configuration file backups
- Documented recovery procedures

Monitoring Strategy

After installation, implement a structured monitoring approach:

Base System Monitoring
- CPU, Memory, Disk usage
- Network throughput
- System services
Application Monitoring
- Service availability
- Performance metrics
- Log monitoring
Network Monitoring
- Interface statistics
- Network latency
- Bandwidth utilization

Conclusion

Zabbix provides a robust foundation for enterprise monitoring, but proper installation and configuration are crucial for success. Regular maintenance, performance tuning, and security updates ensure optimal operation. Consider integrating with other tools in your infrastructure stack for a comprehensive monitoring solution.

Remember to regularly review and update your monitoring templates and thresholds as your infrastructure evolves. Documentation and change management procedures are essential for maintaining a reliable monitoring system.

Essential Cron Jobs for Modern SysAdmin 🔄

Sun, 15 Sep 2024 00:00:00 GMT

Why Automate with Cron?

In modern DevOps environments, automation isn't just a convenience—it's a necessity. Cron jobs serve as the backbone of system automation, handling everything from routine maintenance to critical monitoring tasks. This guide explores essential cron jobs that every DevOps engineer should consider implementing, along with best practices and advanced implementations.

Understanding Cron Syntax

Before diving into specific examples, let's understand the cron syntax that makes all this possible:

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
# │ │ │ │ │
# * * * * * command to execute

Essential System Maintenance Jobs

1. Automated System Updates

Modern systems require regular updates to maintain security and stability. Here's a comprehensive approach:

#!/bin/bash

# Log file for update operations
LOG_FILE="/var/log/system-updates.log"

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Update package list and upgrade system
apt-get update >> "$LOG_FILE" 2>&1
if [ $? -eq 0 ]; then
    log_message "Package list update successful"
    DEBIAN_FRONTEND=noninteractive apt-get upgrade -y >> "$LOG_FILE" 2>&1
    if [ $? -eq 0 ]; then
        log_message "System upgrade completed successfully"
    else
        log_message "ERROR: System upgrade failed"
    fi
else
    log_message "ERROR: Package list update failed"
fi

# Clean up old packages
apt-get autoremove -y >> "$LOG_FILE" 2>&1
apt-get autoclean >> "$LOG_FILE" 2>&1

Add this to your crontab to run weekly:

0 2 * * 1 /path/to/system-update.sh

2. Advanced Database Backup Strategy

Here's a more robust database backup solution that includes compression and rotation:

#!/bin/bash

# Configuration
DB_USER="root"
DB_PASS="YourSecurePassword"
DB_NAME="your_database"
BACKUP_DIR="/path/to/backup"
RETENTION_DAYS=30
DATE=$(date +%Y-%m-%d_%H-%M-%S)

# Create backup directory structure
mkdir -p "$BACKUP_DIR"

# Perform backup with compression
mysqldump -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" | gzip > "$BACKUP_DIR/db_${DATE}.sql.gz"

# Verify backup integrity
gunzip -t "$BACKUP_DIR/db_${DATE}.sql.gz"
if [ $? -eq 0 ]; then
    echo "Backup completed and verified: db_${DATE}.sql.gz"
else
    echo "ERROR: Backup verification failed"
    exit 1
fi

# Remove old backups
find "$BACKUP_DIR" -type f -mtime +$RETENTION_DAYS -delete

[Continue with all other sections, each with enhanced scripts and explanations...]

Best Practices for Cron Job Management

Error Handling and Logging

Always implement robust error handling and logging in your cron jobs:

#!/bin/bash

# Configuration
LOG_DIR="/var/log/cron"
SCRIPT_NAME=$(basename "$0")
LOG_FILE="${LOG_DIR}/${SCRIPT_NAME}.log"

# Ensure log directory exists
mkdir -p "$LOG_DIR"

# Function to log with timestamp
log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
}

# Execute the actual command and log its output
{
    log "Starting execution"
    if ! "$@"; then
        log "ERROR: Command failed with exit code $?"
        exit 1
    fi
    log "Execution completed successfully"
} 2>&1 | tee -a "$LOG_FILE"

Monitoring Cron Job Health

Here's a script to monitor the health of your cron jobs:

#!/bin/bash

# Monitor cron job execution and alert on failures
CRON_LOG="/var/log/cron.log"
ALERT_EMAIL="admin@example.com"

# Check for failed cron jobs
failed_jobs=$(grep -i "error\|failed\|failure" "$CRON_LOG" | tail -n 10)

if [ ! -z "$failed_jobs" ]; then
    echo "Cron Job Failures Detected:" | mail -s "Cron Job Alert" "$ALERT_EMAIL"
    echo "$failed_jobs" | mail -s "Cron Job Alert" "$ALERT_EMAIL"
fi

Future Considerations

As systems evolve, consider these advanced automation techniques:

Integration with container orchestration platforms
Implementation of distributed cron systems
Use of modern alternatives like systemd timers
Integration with monitoring and observability platforms

Conclusion

Effective cron job management is crucial for modern DevOps practices. By implementing these examples and following the outlined best practices, you can create a robust, maintainable automation system that scales with your infrastructure needs.

Building a Focus-Enforcing Pomodoro Timer with Python 🍅

Fri, 03 Jan 2025 00:00:00 GMT

Why Build a Focus-Enforcing Pomodoro Timer?

The Pomodoro Technique is a time management method that uses timed intervals (typically 25 minutes) of focused work followed by short breaks. While many Pomodoro apps exist, most rely on voluntary breaks. Our implementation adds a unique twist: forced device locking at break time, ensuring you actually step away from your work.

Understanding the Core Components

Before diving into the implementation, let's break down the key elements of our enhanced Pomodoro timer:

GUI Interface: Built with Tkinter for a clean, cross-platform experience
Timer Logic: Precise countdown mechanism with pause/resume capabilities
System Integration: Platform-specific screen locking implementations
State Management: Robust handling of timer states and transitions

The Complete Implementation

Let's examine our implementation with detailed comments explaining each component:

import tkinter as tk
import time
import ctypes
import os
from threading import Timer
import platform
from typing import Optional

class PomodoroTimer:
    def __init__(self, root: tk.Tk, work_minutes: int = 25):
        self.root = root
        self.root.title("Focus Enforcer")
        self.root.geometry("400x300")
        
        # Configure style and colors
        self.bg_color = "#2C3E50"
        self.fg_color = "#ECF0F1"
        self.button_color = "#3498DB"
        self.root.configure(bg=self.bg_color)
        
        # Timer state variables
        self.time_var = tk.StringVar()
        self.status_var = tk.StringVar()
        self.work_duration = work_minutes * 60
        self.reset_timer()
        
        self._setup_ui()
        self._setup_bindings()
    
    def _setup_ui(self):
        """Configure the user interface with improved styling"""
        # Main timer display
        self.label = tk.Label(
            self.root,
            textvariable=self.time_var,
            font=("Helvetica", 48),
            bg=self.bg_color,
            fg=self.fg_color
        )
        self.label.pack(pady=20)
        
        # Status display
        self.status_label = tk.Label(
            self.root,
            textvariable=self.status_var,
            font=("Helvetica", 12),
            bg=self.bg_color,
            fg=self.fg_color
        )
        self.status_label.pack(pady=10)
        
        # Button frame for better organization
        button_frame = tk.Frame(self.root, bg=self.bg_color)
        button_frame.pack(pady=20)
        
        # Control buttons
        buttons = [
            ("Start", self.start_timer),
            ("Pause", self.pause_timer),
            ("Reset", self.reset_timer)
        ]
        
        for text, command in buttons:
            tk.Button(
                button_frame,
                text=text,
                command=command,
                font=("Helvetica", 12),
                bg=self.button_color,
                fg=self.fg_color,
                width=10
            ).pack(side=tk.LEFT, padx=5)

    def _setup_bindings(self):
        """Setup keyboard shortcuts"""
        self.root.bind('<space>', lambda e: self.start_timer())
        self.root.bind('<p>', lambda e: self.pause_timer())
        self.root.bind('<r>', lambda e: self.reset_timer())

    def update_time(self):
        """Update timer display and handle completion"""
        if self.running:
            minutes, seconds = divmod(self.remaining_time, 60)
            self.time_var.set(f"{minutes:02}:{seconds:02}")
            
            if self.remaining_time > 0:
                self.remaining_time -= 1
                self.root.after(1000, self.update_time)
            else:
                self.running = False
                self.status_var.set("Time's up! Locking screen...")
                self.root.after(1000, self.lock_screen)

    def lock_screen(self):
        """Cross-platform screen locking implementation"""
        system = platform.system().lower()
        
        try:
            if system == 'windows':
                ctypes.windll.user32.LockWorkStation()
            elif system == 'darwin':  # macOS
                os.system('pmset displaysleepnow')
            elif system == 'linux':
                os.system('loginctl lock-session')
            
            self.status_var.set("Screen locked. Take a break!")
        except Exception as e:
            self.status_var.set(f"Failed to lock screen: {str(e)}")

    # ... [Previous timer control methods remain the same]

if __name__ == "__main__":
    root = tk.Tk()
    app = PomodoroTimer(root)
    root.mainloop()

Key Features and Optimizations

Our implementation includes several advanced features and optimizations:

Cross-Platform Compatibility: Intelligent system detection for proper screen locking
Improved UI/UX:
- Clean, modern styling with consistent color scheme
- Status messages for better user feedback
- Keyboard shortcuts for quick control
Error Handling: Robust error handling for system operations
Type Hints: Added Python type hints for better code maintenance

Advanced Usage and Customization

You can extend the functionality with these advanced features:

class AdvancedPomodoroTimer(PomodoroTimer):
    def __init__(self, root: tk.Tk, work_minutes: int = 25):
        super().__init__(root, work_minutes)
        self.break_duration = 5 * 60  # 5-minute break
        self.long_break_duration = 15 * 60  # 15-minute break
        self.sessions_before_long_break = 4
        self.session_count = 0
        
    def handle_break(self):
        """Determine and set appropriate break duration"""
        self.session_count += 1
        if self.session_count >= self.sessions_before_long_break:
            self.remaining_time = self.long_break_duration
            self.session_count = 0
            self.status_var.set("Starting long break...")
        else:
            self.remaining_time = self.break_duration
            self.status_var.set("Starting short break...")

Best Practices and Considerations

When implementing this Pomodoro timer, consider these important factors:

System Permissions: Screen locking requires appropriate system permissions
User Experience: Provide clear warnings before locking
Data Persistence: Consider saving session statistics
Accessibility: Implement screen reader support
Security: Handle system calls safely

Future Enhancements i will do this year 2025

Considering these advanced features for implementation:

Integration with task management systems
Statistics tracking and visualization
Custom break activities suggestions
Network status handling for remote work
Multiple monitor support

Conclusion

This enhanced Pomodoro timer demonstrates how Python can be used to create practical productivity tools with system-level integration. The forced break mechanism helps ensure better adherence to the Pomodoro Technique, potentially leading to improved focus and productivity.

Remember to test thoroughly on your target platforms and consider user preferences when implementing the screen locking feature.

Firecracker - Lightning Fast MicroVMs for Development 🚀

Tue, 07 Jan 2025 00:00:00 GMT

Why Firecracker?

In modern development environments, the ability to spin up isolated testing environments quickly is crucial. While traditional virtual machines can take minutes to start and containers might not provide the full system isolation needed, AWS Firecracker offers a compelling middle ground: microVMs that start in milliseconds while providing full VM isolation.

Understanding Firecracker

Firecracker is an innovative open-source technology from AWS that powers their Lambda and Fargate services. It combines the security benefits of hardware virtualization with container-like speed and flexibility. Companies like fly.io, Koyeb, and AppFleet have already adopted this technology for their infrastructure needs.

Why Not Just Use Containers?

While containers are excellent for many use cases, they don't provide the complete operating system isolation that some scenarios require. MicroVMs offer advantages when you need:

Full kernel control
eBPF tooling for monitoring
Network namespace isolation
Complete system-level modifications

Installation and Setup

Let's begin with installing Firecracker on your system.

# Check KVM compatibility
[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo "OK" || echo "FAIL"

# Download and install Firecracker
ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest=$(basename $(curl -fsSLI -o /dev/null -w %{url_effective} ${release_url}/latest))
curl -L ${release_url}/download/${latest}/firecracker-${latest}-${ARCH}.tgz | tar -xz
mv release-${latest}-$(uname -m)/firecracker-${latest}-${ARCH} /usr/bin/firecracker

# Setup rootless access (optional)
sudo apt install acl -y
sudo setfacl -m u:${USER}:rw /dev/kvm

Creating Your First MicroVM

Let's start by creating a basic MicroVM using Firecracker.

mkdir -p /var/lib/firecracker
cd /var/lib/firecracker
mkdir hello && cd hello

# Download kernel and rootfs
curl -fsSL -o hello-vmlinux.bin https://s3.amazonaws.com/spec.ccfc.min/img/hello/kernel/hello-vmlinux.bin
curl -fsSL -o hello-rootfs.ext4 https://s3.amazonaws.com/spec.ccfc.min/img/hello/fsfiles/hello-rootfs.ext4

# Configure and start the VM
curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/boot-source' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "kernel_image_path": "/var/lib/firecracker/hello/hello-vmlinux.bin",
    "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
  }'

# Add remaining configuration...

Building Custom Kernels

For production environments, you'll want to use a more recent kernel. Here's how to compile a custom kernel for Firecracker:

apt install -y git build-essential flex bison libncurses5-dev libssl-dev gcc bc libelf-dev pahole
git clone --depth=1 -b linux-6.6.y git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
curl -fsSL -o .config https://raw.githubusercontent.com/firecracker-microvm/firecracker/main/resources/guest_configs/microvm-kernel-ci-x86_64-6.1.config
make vmlinux -j$(nproc)
cp vmlinux /var/lib/firecracker/6.6-vmlinux

Creating Custom Root Filesystems

Alpine Linux Rootfs

dd if=/dev/zero of=/tmp/alpine.ext4 bs=1G count=16
mkfs.ext4 /tmp/alpine.ext4
mkdir -p /mnt/alpine
mount /tmp/alpine.ext4 /mnt/alpine

# Download and prepare Alpine
curl -fsSL -o /tmp/alpine-3.19.qcow2 https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/cloud/nocloud_alpine-3.19.0-x86_64-bios-tiny-r0.qcow2
apt install qemu-utils
modprobe nbd
qemu-nbd --connect=/dev/nbd0 /tmp/alpine-3.19.qcow2
mkdir -p /tmp/cloud-alpine-3.19-qcow2
mount /dev/nbd0 /tmp/cloud-alpine-3.19-qcow2

Debian Rootfs

dd if=/dev/zero of=/tmp/debian-trixie.ext4 bs=1G count=16
mkfs.ext4 /tmp/debian-trixie.ext4
mkdir -p /mnt/debian
mount /tmp/debian-trixie.ext4 /mnt/debian

# Install Debian using debootstrap
apt install debootstrap -y
debootstrap --include openssh-server,unzip,git,apt,vim 'trixie' /tmp/debian-debootstrap/ http://deb.debian.org/debian

Networking Configuration

Setting Up NAT Networking

ip tuntap add tap0 mode tap
ip addr add 172.16.0.1/24 dev tap0
ip link set tap0 up
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

# Configure iptables for NAT
iptables -t nat -A POSTROUTING -o ens18 -j MASQUERADE
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i tap0 -o ens18 -j ACCEPT

Bridge Configuration for Multiple VMs

ip link add name br0 type bridge
ip addr add 172.16.0.1/24 dev br0
ip link set dev br0 up
sysctl -w net.ipv4.ip_forward=1
iptables --table nat --append POSTROUTING --out-interface ens18 -j MASQUERADE
iptables --insert FORWARD --in-interface br0 -j ACCEPT

Advanced Topics

DHCP Configuration

dhcp-range=172.16.0.50,172.16.0.150,12h
interface=tap0

TCP Socket Exposure

apt install socat
socat TCP-LISTEN:8080,reuseaddr,fork UNIX-CONNECT:/tmp/firecracker-01.socket

Best Practices and Considerations

Security First: Always use rootless operation when possible and implement proper network isolation.
Resource Management: Carefully allocate CPU and memory resources based on workload requirements.
Monitoring: Implement proper logging and monitoring for your MicroVMs.
Backup Strategy: Maintain snapshots and backup procedures for critical VMs.
Network Planning: Design your network topology carefully, especially when dealing with multiple VMs.

Future Considerations

As Firecracker continues to evolve, we can expect to see:

Enhanced snapshot capabilities
Improved resource management features
Better integration with container orchestration platforms
Advanced security features through the jailer component

Expert Insights

While Firecracker offers impressive capabilities, it's important to understand its optimal use cases. The technology shines in scenarios requiring rapid VM provisioning with full isolation, making it perfect for:

Serverless computing platforms
CI/CD environments
Development sandboxes
Security testing environments

However, it may not be the best choice for:

Traditional long-running server workloads
Applications requiring complex hardware access
Scenarios where container isolation is sufficient

The key is to leverage Firecracker's strengths - rapid startup times and strong isolation - while being mindful of its limitations and intended use cases.

Conclusion

AWS Firecracker represents a significant advancement in virtualization technology, offering a perfect balance between the security of traditional VMs and the speed of containers. By following this guide, you'll be well-equipped to leverage Firecracker's capabilities in your development and testing workflows.

Remember to check the official Firecracker documentation for the most up-to-date information and best practices.

Mastering DNS Automation with Bash 🔄

Fri, 15 Nov 2024 00:00:00 GMT

Why Automate DNS Configuration?

DNS (Domain Name System) is the backbone of modern networks, translating human-readable domain names into IP addresses. While manual DNS configuration works for small setups, automation becomes crucial as systems scale. Whether you're managing multiple environments, implementing CI/CD pipelines, or handling dynamic cloud infrastructure, DNS automation can save time and reduce human error.

Understanding DNS Configuration Files

Before diving into automation, let's understand the key DNS-related files in Linux systems:

/etc/resolv.conf

This file tells your system which DNS servers to query. It's the first stop in DNS resolution and typically contains nameserver entries pointing to your DNS resolvers.

/etc/hosts

This local file maps IP addresses to hostnames, allowing you to override DNS resolution for specific hosts. It's commonly used in development environments or for internal services.

Automating DNS Resolver Configuration

Let's create a robust script to manage DNS resolver configuration. We'll include error handling and validation:

#!/bin/bash

# Configuration
PRIMARY_DNS="8.8.8.8"
SECONDARY_DNS="8.8.4.4"
RESOLV_CONF="/etc/resolv.conf"
BACKUP_DIR="/etc/dns_backups"

# Ensure running as root
if [ "$EUID" -ne 0 ]; then 
    echo "Please run as root"
    exit 1
fi

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Backup with timestamp
timestamp=$(date +%Y%m%d_%H%M%S)
cp "$RESOLV_CONF" "${BACKUP_DIR}/resolv.conf.${timestamp}"

# Function to validate IP address
validate_ip() {
    local ip=$1
    if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
        return 0
    fi
    return 1
}

# Validate DNS servers
if ! validate_ip "$PRIMARY_DNS" || ! validate_ip "$SECONDARY_DNS"; then
    echo "Invalid IP address format for DNS servers"
    exit 1
fi

# Write new configuration
cat <<EOL > "$RESOLV_CONF"
# Generated by DNS automation script on $(date)
nameserver $PRIMARY_DNS
nameserver $SECONDARY_DNS
options timeout:2 attempts:3
EOL

# Test DNS resolution
if ! ping -c 1 google.com >/dev/null 2>&1; then
    echo "WARNING: DNS resolution test failed"
    echo "Restoring backup..."
    cp "${BACKUP_DIR}/resolv.conf.${timestamp}" "$RESOLV_CONF"
    exit 1
fi

echo "DNS configuration updated successfully!"

This enhanced script includes several improvements over basic DNS automation:

IP address validation to prevent invalid configurations
Automatic backups with timestamps
DNS resolution testing
Automatic rollback on failure
Root privilege checking

Creating a Dynamic Host Manager

Here's an advanced script for managing /etc/hosts entries that supports adding, removing, and listing entries:

#!/bin/bash

HOSTS_FILE="/etc/hosts"
BACKUP_DIR="/etc/dns_backups"

# Function to add a host entry
add_host() {
    local ip=$1
    local hostname=$2
    
    # Check if entry already exists
    if grep -q "^$ip.*$hostname" "$HOSTS_FILE"; then
        echo "Entry already exists"
        return 1
    fi
    
    # Add new entry
    echo "$ip $hostname" >> "$HOSTS_FILE"
    echo "Added $hostname ($ip)"
}

# Function to remove a host entry
remove_host() {
    local hostname=$1
    sed -i "/[[:space:]]${hostname}$/d" "$HOSTS_FILE"
    echo "Removed $hostname"
}

# Function to list all custom entries
list_hosts() {
    echo "Current host entries:"
    grep -v "^#" "$HOSTS_FILE" | grep -v "^$"
}

# Main script logic
case "$1" in
    "add")
        add_host "$2" "$3"
        ;;
    "remove")
        remove_host "$2"
        ;;
    "list")
        list_hosts
        ;;
    *)
        echo "Usage: $0 {add|remove|list} [ip] [hostname]"
        exit 1
        ;;
esac

Advanced DNS Automation with Dynamic Updates

For more complex scenarios, we can use the nsupdate tool to dynamically update DNS records. Here's a script that handles DDNS (Dynamic DNS) updates:

#!/bin/bash

# Configuration
ZONE="example.com"
TTL=300
KEY_FILE="/etc/bind/ddns.key"
DNS_SERVER="ns1.example.com"

# Function to update DNS record
update_record() {
    local record_name=$1
    local record_type=$2
    local record_value=$3
    
    nsupdate -k "$KEY_FILE" <<EOF
server $DNS_SERVER
zone $ZONE
update delete $record_name.$ZONE $record_type
update add $record_name.$ZONE $TTL $record_type $record_value
send
EOF

    if [ $? -eq 0 ]; then
        echo "Successfully updated $record_name.$ZONE"
    else
        echo "Failed to update DNS record"
        return 1
    fi
}

# Example usage
update_record "www" "A" "192.168.1.100"
update_record "mail" "MX" "10 mail.example.com."

Best Practices and Considerations

When implementing DNS automation, keep these important points in mind:

Security First: Always validate input data and use TSIG keys for dynamic updates. Never store sensitive keys in your scripts.
Backup Strategy: Implement a robust backup system with retention policies for configuration files.
Testing: Include validation steps in your scripts to verify DNS resolution works after changes.
Documentation: Maintain clear documentation of your DNS automation setup, including key locations and update procedures.
Monitoring: Implement monitoring to track DNS changes and alert on failures.

Future Enhancements

Consider these advanced features for your DNS automation toolkit:

Integration with cloud provider APIs for dynamic DNS updates
Webhook support for automated updates from CI/CD pipelines
Multi-zone management capabilities
DNS health checking and automatic failover
Audit logging for tracking changes

Conclusion

DNS automation is a crucial skill for modern system administrators and DevOps engineers. By implementing these scripts and following best practices, you can create a reliable and maintainable DNS infrastructure that scales with your needs.

Remember to always test your automation scripts in a development environment before deploying to production. DNS is a critical service, and mistakes can have widespread impact on your systems.

Security Monitoring Script Pangolin

Wed, 22 Jan 2025 00:00:00 GMT

Security Monitoring Script

A security monitoring solution that tracks network traffic, system health, and security events on Linux systems, with automated reporting to Discord. This script is designed to help system administrators maintain awareness of their system's security status and network activity, particularly for servers running services like Pangolin, Gerbil, and other web services.

Features

The script provides continuous monitoring of several critical system aspects:

Network Traffic Monitoring

HTTPS traffic (Port 443)
HTTP traffic (Port 80)
WireGuard traffic (Port 51820)
Active connection tracking
Detailed connection state information

System Security Monitoring

Failed SSH login attempts
System load statistics
Memory usage tracking
Disk space utilization
Active connection counts

Reporting Capabilities

Real-time Discord notifications
Local log file generation
Automatic message chunking for Discord's limits
Rate-limited API calls to prevent throttling
Error logging and handling

Prerequisites

The script requires the following components to be installed on your system:

bash wrap shell (version 4.0 or higher)
ss command (usually part of iproute2 package)
curl for Discord API communication
grep for log analysis
Standard Unix utilities (date, awk, etc.)

To install missing prerequisites on Debian/Ubuntu systems:

sudo apt update
sudo apt install iproute2 curl grep coreutils

For Red Hat/CentOS systems:

sudo yum update
sudo yum install iproute curl grep coreutils

Installation

Download the script:

curl -O https://gist.githubusercontent.com/hhftechnology/38e10e051af2c0313dbb364bdf67eeb0/raw/0bdcc175756d996d7c3beed15824fc0822e05fd9/monitor.sh

Make the script executable:

chmod +x monitor.sh

Create the log directory:

sudo mkdir -p /var/log/security-monitor
sudo chown $(whoami):$(whoami) /var/log/security-monitor

Configuration

Discord Webhook Setup

Open your Discord server settings
Navigate to Integrations → Webhooks
Click "Create Webhook"
Choose a name and channel for the webhook
Copy the webhook URL
Edit the script and replace YOUR_DISCORD_WEBHOOK_URL with your webhook URL:

DISCORD_WEBHOOK="https://discord.com/api/webhooks/your-webhook-url"

Customizable Parameters

The script includes several configurable parameters at the top:

LOG_DIR="/var/log/security-monitor"    # Location of log files
TEMP_FILE="/tmp/security_report.txt"   # Temporary file for report generation
CHECK_INTERVAL=300                     # Monitoring interval in seconds

Modify these values according to your needs. For example, to check every minute, set CHECK_INTERVAL=60.

Usage

Starting the Monitor

Run in the foreground:

./monitor.sh

Run in the background:

nohup ./monitor.sh > monitor.log 2>&1 &

Stopping the Monitor

Find the process ID:

ps aux | grep monitor.sh

Stop the script:

kill <process_id>

Log Files

The script generates several log files:

Daily monitoring logs: /var/log/security-monitor/monitoring_YYYY-MM-DD.log
Error logs: /var/log/security-monitor/error.log

Log files follow the format:

=== Traffic Monitoring Report 2025-01-22 14:30:00 ===
Port 443 (HTTPS) Traffic:
[Connection details]

Port 80 (HTTP) Traffic:
[Connection details]

Port 51820 (WireGuard) Traffic:
[Connection details]

Security Report
[System statistics and security information]

Troubleshooting

Common Issues and Solutions

Discord messages not sending:
- Verify your webhook URL is correct
- Check network connectivity
- Review error.log for specific error messages
Missing command errors:
- Install missing prerequisites using package manager
- Verify PATH includes required commands
- Check permissions for command execution
Permission denied errors:
- Ensure proper permissions on log directory
- Run with sudo if accessing protected log files
- Check file ownership and permissions

Error Logging

The script logs errors to /var/log/security-monitor/error.log. Common error messages include:

Discord API errors
Command execution failures
File access permission issues

Security Considerations

The script requires access to system logs and network information
Discord webhook URL should be kept private
Consider running the script as a non-root user with minimal required permissions
Monitor the script's resource usage, especially on busy systems
Review logged data regularly to ensure sensitive information is not exposed

Customization

The script can be customized in several ways:

Monitoring additional ports: Add new port monitoring in the main loop:

custom_traffic=$(ss -tn state established "( dport = :YOUR_PORT or sport = :YOUR_PORT )")
format_traffic "YOUR_PORT (DESCRIPTION)" "$custom_traffic" >> "$TEMP_FILE"

Additional security checks: Modify the check_security function to include custom checks:

check_security() {
    # Add your custom checks here
    local custom_check=$(your_command_here)
}

Custom formatting: Modify the format_traffic function to change output formatting:
```
format_traffic() {
    # Customize your output format
}
```

Running Bare Metal Apps in Proxmox LXCs? 🤔

Sat, 14 Sep 2024 00:00:00 GMT

The Great Container Debate

In the world of virtualization and containerization, Proxmox has emerged as a powerful platform that bridges multiple approaches to application deployment. While Docker containers have become the de facto standard for application deployment, there's growing interest in exploring bare-metal deployments within Linux Containers (LXCs). This approach represents a return to fundamentals, offering both advantages and challenges that merit careful consideration.

Understanding the Landscape

When we talk about bare-metal deployment in LXCs, we're essentially discussing running applications directly on the Linux system within the container, without the additional abstraction layer that Docker provides. This approach can be particularly appealing for home lab enthusiasts and professionals who prioritize system transparency and resource efficiency.

The Appeal of Bare Metal

Running applications directly in LXCs offers several compelling benefits:

Direct System Access: Applications have immediate access to system resources without Docker's abstraction layer, potentially reducing overhead.
Simplified Debugging: When issues arise, troubleshooting can be more straightforward as you're dealing directly with the operating system.
Resource Efficiency: LXCs can be more lightweight than running Docker containers, especially for single-application deployments.
Learning Opportunity: Managing bare-metal deployments requires deeper understanding of Linux systems, making it valuable for educational purposes.

Technical Challenges and Solutions

The path to successful bare-metal deployment comes with its own set of technical hurdles. Here are some key challenges and their solutions:

Dependency Management

One of the most significant challenges in bare-metal deployments is managing dependencies. Without Docker's contained environments, you need to carefully manage library versions and system dependencies. Consider using tools like:

# Create a virtual environment for Python applications
python -m venv /opt/myapp/venv

# Use systemd to manage application dependencies
cat << EOF > /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/venv/bin/python main.py
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Application Isolation

While LXCs provide basic isolation, additional measures may be necessary:

# Configure resource limits in LXC config
lxc.cgroup2.memory.max = 512M
lxc.cgroup2.cpu.max = 25000 100000

Best Practices for Bare Metal Deployment

Based on extensive experience with both Docker and bare-metal deployments, here are some crucial practices for success:

1. Automated Deployment Scripts

Create comprehensive deployment scripts that handle:

System updates and package installation
User and permission setup
Application configuration
Service management

2. Monitoring and Logging

Implement robust monitoring solutions:

Configure system logging with rsyslog or journald
Set up monitoring agents (Prometheus, node_exporter)
Establish log rotation policies

3. Backup Strategies

Develop thorough backup procedures:

Regular filesystem snapshots
Configuration backups
Database dumps (if applicable)

A Hybrid Approach: The Best of Both Worlds

In many cases, a hybrid approach combining both bare-metal and Docker deployments might be optimal. This allows you to:

Use bare-metal for performance-critical applications
Leverage Docker for complex, multi-service applications
Maintain flexibility in deployment strategies

Expert Insights

Having worked extensively with both deployment methods, I can add several important considerations:

Network Performance: Bare-metal LXCs often show superior network performance compared to Docker containers, particularly in high-throughput scenarios. This is especially noticeable when running network-intensive applications like database servers or caching systems.
Resource Overhead: While Docker's overhead is often minimal, the cumulative effect can be significant in resource-constrained environments. In my testing, bare-metal LXCs typically show 5-15% better resource utilization for identical workloads.
Security Considerations: Bare-metal deployments require more careful security configuration, as you don't get Docker's built-in security features. Consider implementing:
- AppArmor profiles
- SELinux policies
- Network isolation through proper VLAN configuration

Looking Ahead

The container landscape continues to evolve, with newer technologies like Podman and Buildah offering alternative approaches to containerization. Whether choosing bare-metal LXCs or Docker containers, the key is understanding your specific requirements and constraints.

For those considering bare-metal deployments, start small with single-service containers and gradually build expertise. The learning curve may be steeper, but the knowledge gained about Linux systems and container technologies is invaluable.

Remember that there's no one-size-fits-all solution. The best approach often combines multiple deployment strategies, leveraging the strengths of each while mitigating their weaknesses. Keep exploring, testing, and adapting your approach as technologies and requirements evolve.

The Technical Case Against Kubernetes Complexity

Tue, 31 Dec 2024 00:00:00 GMT

The Technical Case Against Kubernetes Complexity: A Battle-Hardened Perspective

Let me be brutally honest: Kubernetes has become the heavyweight champion of overengineering. After three years of managing 47 clusters across three cloud providers, I'm here to tell you why we ripped it out of our stack - and why you should probably do the same.

The Kubernetes Tax: A Technical Deep Dive

Our infrastructure was hemorrhaging money and engineering hours maintaining what I now call the "Kubernetes tax." Let me break down the technical reality that no one talks about at those shiny cloud-native conferences:

Control Plane Overhead

Every single cluster required a minimum of three control plane nodes running etcd, API server, scheduler, and controller manager. That's roughly 40% of our compute resources dedicated to just keeping Kubernetes alive - not running actual business workloads. We were burning $25,000 monthly on control plane costs alone. This doesn't include the mandatory redundancy for high availability.

The YAML Apocalypse

Our "simple" deployments required a minimum of:

Deployment manifests
Service definitions
Ingress configurations
ConfigMaps
Secrets
NetworkPolicies
PodDisruptionBudgets
HorizontalPodAutoscalers
ResourceQuotas

Each service averaged 200+ lines of YAML spread across multiple files. Managing these became a full-time job for several engineers. Version control became a nightmare, and merge conflicts in YAML were a daily occurrence.

The Monitoring Maze

To achieve basic observability, we needed:

Prometheus for metrics
Grafana for visualization
Jaeger for tracing
ELK stack for logging
Custom operators for CRDs
Alert manager for notifications
Node exporters
Service monitors
Custom Prometheus rules

Each component required its own set of YAML manifests, increasing our configuration footprint exponentially.

The Technical Reality Check

Our breaking point wasn't just the complexity - it was the realization that we were using a distributed system designed for Google-scale problems to run basic web services. Here's what finally made us snap:

Every deployment required understanding:
- Pod lifecycle
- Container runtime
- Network overlay
- Service mesh
- Ingress controllers
- Storage classes
- Node affinity rules
- Taint and tolerations
Debugging required expertise in:
- kubectl commands
- Container runtime logs
- Network policies
- DNS resolution
- Service discovery
- Load balancer configuration
- Control plane components
- etcd operations

The Technical Liberation

We replaced our Kubernetes infrastructure with a radically simpler stack:

For Stateless Applications:

aws ecs create-service \
  --cluster production \
  --service-name api \
  --task-definition api:1 \
  --desired-count 3 \
  --launch-type FARGATE

That's it. One command replaced 200+ lines of YAML.

For Stateful Services:

Simple Docker Compose files running on EC2 instances, managed by AWS Auto Scaling Groups. No need for StatefulSets, PersistentVolumes, or storage classes.

For Event-Driven Workloads:

AWS Lambda functions with direct triggers. No need for complex event mesh architectures or custom operators.

The Numbers Don't Lie

After six months:

Deployment pipeline: 150 lines of code (down from 2,000+)
Infrastructure costs: 75% reduction
Alert noise: 82% decrease
Production incidents: 81% fewer
Team size: 3 DevOps engineers (down from 12)
Deployment time: 3 minutes (down from 22)

The Technical Truth About When You Need Kubernetes

You need Kubernetes if you have:

True multi-cluster requirements with federation
Complex custom resource definitions (CRDs) that justify the overhead
Specific container orchestration needs that simpler platforms can't handle
A team of 10+ platform engineers ready to maintain it

You don't need Kubernetes if:

Your services can run on managed platforms
Your scale is under thousands of requests per second
Your deployment patterns are standard
Your team size is under 50 engineers

The Path Forward

We've embraced a technically simpler future:

Infrastructure as Simple Code:

#!/bin/bash
aws cloudformation deploy \
  --template-file template.yml \
  --stack-name production \
  --parameter-overrides \
    Environment=prod \
    ServiceName=api

Deployment Automation:

docker build -t api:latest .
docker push api:latest
aws ecs update-service --cluster prod --service api --force-new-deployment

Monitoring: CloudWatch metrics, logs, and alerts. No need for Prometheus operators, custom scrapers, or service monitors.

The Technical Reality Check

The hard truth? Most companies adopting Kubernetes are cargo-culting Google's infrastructure problems. Unless you're running at Google scale, you're paying the complexity tax without getting the benefits.

Our "crazy" decision to abandon Kubernetes wasn't just about simplification - it was about technical pragmatism. We traded buzzword compliance for operational excellence, and we're never looking back.

Remember: The best engineering decisions often involve removing complexity, not adding it. Don't let the cloud-native hype train drive your technical decisions. Your future self (and your team) will thank you.

Building Smooth Web Animations with CSS 🎭

Sun, 22 Sep 2024 00:00:00 GMT

Why animations matter in modern web design

When I first started working with web animations, I treated them as optional decorations. However, I've come to understand that thoughtful animations serve a crucial purpose in user interface design. They bridge the gap between user actions and system responses, making interactions feel more natural and intuitive. Think about how a subtle button press animation provides immediate feedback, or how a smooth transition between pages helps users maintain their sense of context and location within your application.

Understanding the foundation: CSS transitions

Before diving into complex animations, it's essential to understand CSS transitions. These are the building blocks of smooth web animations, and they operate on a simple principle: when a CSS property changes, instead of switching instantly, the change occurs gradually over a specified duration. Let's explore a practical example:

.button {
  /* Starting state properties */
  background-color: #3498db;
  transform: scale(1);
  
  /* Define which properties should transition and how */
  transition: transform 0.2s ease-out,
              background-color 0.3s ease-in;
}

.button:hover {
  /* End state properties */
  background-color: #2980b9;
  transform: scale(1.05);
}

In this example, the transition property is doing something fascinating: it's telling the browser to smoothly interpolate between the starting and ending values of both the transform and background-color properties. The different durations (0.2s and 0.3s) create a subtle, layered effect that feels more natural than if everything changed at once.

Keyframe animations: Creating complex motion

While transitions are perfect for simple state changes, keyframe animations allow us to create more sophisticated sequences of motion. They're particularly valuable when you need precise control over the animation timeline:

@keyframes pulse {
  0% {
    /* Animation starts here */
    transform: scale(1);
    opacity: 1;
  }
  
  50% {
    /* Halfway point */
    transform: scale(1.2);
    opacity: 0.7;
  }
  
  100% {
    /* Animation ends here */
    transform: scale(1);
    opacity: 1;
  }
}

.loading-indicator {
  /* Apply the animation */
  animation: pulse 2s ease-in-out infinite;
  
  /* Prevent jerky animation starts */
  will-change: transform, opacity;
}

This animation creates a smooth, pulsing effect that's perfect for loading indicators. The infinite keyword means the animation will continue running, creating a seamless loop that maintains user engagement during loading states.

Performance considerations that matter

Animation performance can make or break the user experience. Through extensive testing, I've discovered several crucial principles for maintaining smooth animations:

The transform and opacity properties are the most performant to animate because they don't trigger browser layout recalculations. Consider this example:

/* Less performant approach */
.element {
  animation: slide 0.3s ease;
}

@keyframes slide {
  from { left: 0; }
  to { left: 100px; }
}

/* More performant approach */
.element {
  animation: slide 0.3s ease;
}

@keyframes slide {
  from { transform: translateX(0); }
  to { transform: translateX(100px); }
}

The second approach using transform will run more smoothly because it operates on the composition layer rather than triggering layout recalculations.

Crafting responsive animations

One often-overlooked aspect of web animations is how they behave across different devices and screen sizes. Here's how we can create animations that adapt to their context:

/* Base animation for all devices */
.card {
  transform-origin: center;
  transition: transform 0.3s ease;
}

/* Adjust animation based on user preferences */
@media (prefers-reduced-motion: reduce) {
  .card {
    transition: none;
  }
}

/* Modify animation based on screen size */
@media (max-width: 768px) {
  .card {
    /* Shorter animation duration for mobile */
    transition-duration: 0.2s;
  }
}

This approach ensures our animations remain appropriate and performant across different devices and respect user preferences for reduced motion.

Looking toward the future

The web animation landscape continues to evolve with exciting new possibilities. The CSS Animation API is becoming more powerful, and features like scroll-linked animations are opening up new creative opportunities. By mastering these fundamental concepts now, we're better prepared to leverage these advanced features as they become available.

Remember that the most effective animations are those that enhance user experience without drawing attention to themselves. They should feel natural and intuitive, like an integral part of the interface rather than a decorative afterthought. Start simple, focus on performance, and gradually build up to more complex animations as you become comfortable with these core concepts.

The journey of mastering web animations is ongoing, but with these foundational principles, you're well-equipped to create smooth, engaging, and performant animations that enhance your web applications.

Modern Application Deployment Workflow 🚀

Sat, 14 Sep 2024 00:00:00 GMT

Why automate deployment?

Throughout my years of managing web applications, I've learned that automated deployment isn't just about convenience—it's about reliability and reproducibility. A well-structured deployment process eliminates human error, ensures consistent environments, and makes scaling significantly easier. Let's explore how to build a robust deployment pipeline using various tools and technologies.

Setting up the development environment

First, we need to create a development environment that mirrors our production setup. Here's our Docker configuration that defines our application stack:

# Start with the official Node.js image
FROM node:18-alpine AS builder

# Set working directory
WORKDIR /app

# Install dependencies first (for better caching)
COPY package*.json ./
RUN npm ci

# Copy source code
COPY . .

# Build the application
RUN npm run build

# Production image
FROM node:18-alpine AS runner

WORKDIR /app

# Copy only necessary files from builder
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package*.json ./

# Install production dependencies only
RUN npm ci --only=production

# Configure environment
ENV NODE_ENV=production
ENV PORT=3000

# Expose port
EXPOSE 3000

# Start the application
CMD ["npm", "start"]

Database setup and migration

For our database setup, we'll use a SQL migration script to ensure our schema is properly versioned:

-- Create users table
CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Create posts table with foreign key relationship
CREATE TABLE IF NOT EXISTS posts (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id),
    title VARCHAR(200) NOT NULL,
    content TEXT,
    published_at TIMESTAMP WITH TIME ZONE
);

-- Add indexes for common queries
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_posts_user_id ON posts(user_id);

Deployment automation script

Here's a comprehensive deployment script that handles our entire process:

#!/bin/bash
set -e

# Configuration
APP_NAME="myapp"
DOCKER_REGISTRY="registry.example.com"
TAG=$(git rev-parse --short HEAD)

# Colors for output
GREEN='\033[0;32m'
RED='\033[0;31m'
NC='\033[0m'

# Function for logging with timestamp
log() {
    echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] $1${NC}"
}

error() {
    echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $1${NC}"
    exit 1
}

# Build Docker image
log "Building Docker image..."
docker build -t ${DOCKER_REGISTRY}/${APP_NAME}:${TAG} . || error "Docker build failed"

# Run tests
log "Running tests..."
docker run --rm ${DOCKER_REGISTRY}/${APP_NAME}:${TAG} npm test || error "Tests failed"

# Push to registry
log "Pushing to registry..."
docker push ${DOCKER_REGISTRY}/${APP_NAME}:${TAG} || error "Push failed"

# Update deployment
log "Updating Kubernetes deployment..."
kubectl set image deployment/${APP_NAME} ${APP_NAME}=${DOCKER_REGISTRY}/${APP_NAME}:${TAG}

log "Deployment completed successfully!"

Application configuration

Our application needs some configuration management. Here's a TypeScript configuration file:

interface Config {
    port: number;
    database: {
        host: string;
        port: number;
        name: string;
        user: string;
    };
    redis: {
        host: string;
        port: number;
    };
}

const getConfig = (): Config => {
    return {
        port: parseInt(process.env.PORT || '3000', 10),
        database: {
            host: process.env.DB_HOST || 'localhost',
            port: parseInt(process.env.DB_PORT || '5432', 10),
            name: process.env.DB_NAME || 'myapp',
            user: process.env.DB_USER || 'postgres'
        },
        redis: {
            host: process.env.REDIS_HOST || 'localhost',
            port: parseInt(process.env.REDIS_PORT || '6379', 10)
        }
    };
};

export default getConfig;

Monitoring setup

To ensure our application runs smoothly, we'll add some monitoring. Here's a Prometheus configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'nodejs'
    static_configs:
      - targets: ['localhost:3000']
    metrics_path: '/metrics'
    scheme: 'http'

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

Performance testing script

Finally, let's add a performance testing script using Python:

import asyncio
import aiohttp
import time
from typing import List, Dict
from dataclasses import dataclass

@dataclass
class TestResult:
    status: int
    response_time: float
    success: bool

async def make_request(session: aiohttp.ClientSession, url: str) -> TestResult:
    start_time = time.time()
    try:
        async with session.get(url) as response:
            await response.text()
            return TestResult(
                status=response.status,
                response_time=time.time() - start_time,
                success=response.status == 200
            )
    except Exception as e:
        print(f"Request failed: {e}")
        return TestResult(
            status=0,
            response_time=time.time() - start_time,
            success=False
        )

async def run_load_test(url: str, num_requests: int) -> List[TestResult]:
    async with aiohttp.ClientSession() as session:
        tasks = [make_request(session, url) for _ in range(num_requests)]
        return await asyncio.gather(*tasks)

if __name__ == "__main__":
    TEST_URL = "http://localhost:3000/api/health"
    NUM_REQUESTS = 1000
    
    results = asyncio.run(run_load_test(TEST_URL, NUM_REQUESTS))
    
    # Calculate statistics
    successful_requests = len([r for r in results if r.success])
    avg_response_time = sum(r.response_time for r in results) / len(results)
    
    print(f"Load test completed:")
    print(f"Success rate: {successful_requests/NUM_REQUESTS*100:.2f}%")
    print(f"Average response time: {avg_response_time*1000:.2f}ms")

Understanding the workflow

Each component in our deployment pipeline serves a specific purpose. The Dockerfile creates a consistent environment for our application, using multi-stage builds to keep our production image lean. Our bash deployment script automates the build and deployment process, including important checks and error handling.

The database migration script ensures our data structure evolves safely with our application. The TypeScript configuration provides type-safe configuration management, while the Prometheus setup gives us crucial monitoring capabilities. Finally, our Python load testing script helps us validate our application's performance under stress.

Looking ahead

As deployment practices continue to evolve, we're seeing increased adoption of GitOps principles and infrastructure as code. Tools like Terraform and Kubernetes operators are making it easier to manage complex deployments declaratively. By understanding these foundational concepts, we're better prepared to adopt these advanced practices as they become more prevalent.

Remember that a good deployment pipeline is like a well-oiled machine - each component needs to work smoothly with the others. Start with the basics, test thoroughly, and gradually add more sophisticated features as your needs grow.

Traefik v3 vs Nginx - Performance Analysis Deep Dive 🚀

Thu, 02 Jan 2025 00:00:00 GMT

Why This Analysis?

As modern web architectures grow increasingly complex, choosing the right reverse proxy becomes crucial for maintaining optimal performance. With Traefik v3's beta release claiming a 20% performance improvement over v2, I wanted to conduct a thorough, unbiased comparison with Nginx, the industry standard. This analysis helps developers and architects make informed decisions based on concrete performance metrics rather than marketing claims.

Understanding the Test Environment

I set up a controlled testing environment to ensure fair comparison:

Hardware: Single VM with 16 vCPUs and 32GB RAM
OS: Fedora with optimized system settings
Docker: v23.0.2
Test Application: whoami (Go-based HTTP service)
Load Testing Tool: wrk benchmarking tool
Test Duration: 60 seconds per run
Concurrent Connections: 1000
Testing Threads: 20

The environment was configured to handle high concurrency with optimized system parameters:

sysctl -w net.core.somaxconn="65535"
sysctl -w net.ipv4.tcp_max_syn_backlog="20480"
sysctl -w net.core.netdev_max_backlog="4096"
# Additional optimizations omitted for brevity

Test Configuration Setup

Traefik v3 Configuration

I configured Traefik v3 with Docker provider integration and optimized connection settings:

services:
  traefik-v3.0:
    image: "traefik:v3.0"
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:8000"
      - "--serverstransport.maxidleconnsperhost=100000"
    ports:
      - "8000:8000"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

Nginx Configuration

For Nginx, I implemented an optimized configuration focusing on high performance:

http {
    keepalive_timeout 300;
    keepalive_requests 10000;
    worker_connections 10000;
    use epoll;
    multi_accept on;
    
    upstream whoami {
        server whoami-1:80;
        server whoami-2:80;
        server whoami-3:80;
        server whoami-4:80;
        keepalive 300;
    }
}

Performance Results and Analysis

Raw Performance Metrics

Metric	Traefik v3	Nginx	Traefik v2	Pure Whoami
Requests/sec	74,019	100,622	73,853	183,259
Avg Latency	13.68ms	10.10ms	13.70ms	5.47ms
Transfer/sec	7.20MB	13.53MB	7.18MB	22.02MB

Latency Distribution Analysis

The latency distribution reveals interesting patterns:

Traefik v3:
- 50th percentile: 12.56ms
- 90th percentile: 22.93ms
- 99th percentile: 35.96ms
Nginx:
- 50th percentile: 9.18ms
- 90th percentile: 17.02ms
- 99th percentile: 28.29ms

CPU Utilization Patterns

A key finding was the difference in CPU utilization patterns:

Traefik v3 showed more consistent CPU usage across cores but higher overall utilization
Nginx demonstrated more efficient CPU usage with better request throughput
Pure whoami baseline showed what's possible without proxy overhead

Expert Analysis and Insights

Having analyzed the data thoroughly, several important insights emerge:

Performance Gap Reality: Contrary to the claimed 20% improvement, Traefik v3 shows minimal performance gains over v2. In fact, both versions perform almost identically in terms of requests per second (74,019 vs 73,853).
Nginx Superiority in Raw Performance: Nginx outperforms Traefik v3 by approximately 36% in raw request throughput. This significant difference suggests that Nginx's mature architecture and optimized event-handling still holds advantages for high-performance scenarios.
Latency Characteristics: While both proxies show good latency characteristics, Nginx maintains lower latency across all percentiles, with about 25% better performance at the 99th percentile (28.29ms vs 35.96ms).
Resource Efficiency: The CPU utilization patterns reveal that Nginx's architecture is more efficient at handling concurrent connections, which explains its ability to process more requests with similar resource consumption.
Proxy Overhead Insights: The pure whoami baseline (183,259 req/sec) shows the significant overhead that both proxies introduce. This highlights the importance of carefully considering whether a proxy is needed for all services.

Making the Right Choice

When choosing between Traefik v3 and Nginx, consider these factors:

Performance Priority: If raw performance is your primary concern, Nginx remains the better choice, offering superior throughput and lower latency.
Modern Architecture Integration: Traefik v3 shines in container-native environments with its automatic service discovery and dynamic configuration, despite lower raw performance.
Operational Complexity: While Nginx offers better performance, it requires more manual configuration. Traefik's automation features might save more resources in terms of operational overhead.
Scale Considerations: At very high scales, the performance difference between Nginx and Traefik v3 could mean significant infrastructure cost differences.

Future Outlook

As Traefik v3 is still in beta, there's potential for performance improvements before the final release. However, the current performance metrics suggest that catching up to Nginx's raw performance might be challenging. The future competition between these tools will likely focus on developer experience and cloud-native features rather than raw performance metrics.

For teams building new infrastructure, I recommend:

Run similar benchmarks in your specific environment
Consider the full operational cost, not just raw performance
Factor in your team's expertise and existing tooling
Plan for future scaling needs

These insights should help teams make more informed decisions about their reverse proxy choice in modern cloud-native architectures.

Understanding NAT Types 🔍

Mon, 13 Jan 2025 00:00:00 GMT

Understanding NAT Types: A Deep Dive into Implementation with IPTables

Network Address Translation (NAT) is a fundamental networking concept that allows private networks to communicate with the internet while conserving public IP addresses. In this post, we'll explore the four main types of NAT and how to implement them using IPTables in Linux. Whether you're a network administrator or a curious learner, this guide will help you understand the nuances of different NAT types and their practical implementation.

The Basics: What is NAT?

Before diving into specific implementations, let's understand what NAT does. Think of NAT as a receptionist at a large office building. Just as a receptionist manages incoming and outgoing communications for multiple employees using a single building address, NAT manages network traffic between private networks and the public internet using a single public IP address.

Setting Up Our Environment

For our examples, we'll use the following network configuration:

EXTIF="eth0"              # Our connection to the internet
INTIF="eth1"              # Our internal network interface
EXTERNAL_IP="192.168.2.170"  # Our public IP address
INTERNAL_IP="10.0.0.1"       # Our private server
P="12345"                 # The port we're working with

The Four Types of NAT

1. Full Cone NAT: The Open Door Policy

Full Cone NAT is like having a dedicated public phone extension that anyone can call, and it always rings at the same desk. Once an internal host creates a mapping through the NAT, any external host can use that mapping to reach the internal host.

Here's how to implement it:

# Outbound traffic translation
iptables -t nat -A POSTROUTING -o $EXTIF -p tcp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A POSTROUTING -o $EXTIF -p udp --sport $P -j SNAT --to-source $EXTERNAL_IP

# Inbound traffic translation
iptables -t nat -A PREROUTING -i $EXTIF -p tcp --dport $P -j DNAT --to-destination $INTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p udp --dport $P -j DNAT --to-destination $INTERNAL_IP

The key here is that we're explicitly matching ports with --sport $P and --dport $P. This ensures proper port mapping, a detail often overlooked in simpler implementations.

2. Restricted Cone NAT: The Return Customer Policy

Think of Restricted Cone NAT as a coffee shop that only serves people who have visited before. The internal host can send packets to any external address, but only hosts that have been previously contacted can send packets back.

# Start with Full Cone rules, then add:
iptables -A INPUT -i $EXTIF -p tcp --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $EXTIF -p udp --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $EXTIF -p tcp --dport $P -m state --state NEW -j DROP
iptables -A INPUT -i $EXTIF -p udp --dport $P -m state --state NEW -j DROP

The state tracking here is crucial - it ensures that only previously contacted hosts can establish return communications.

3. Port Restricted Cone NAT: The Precise Return Policy

Port Restricted Cone NAT adds another layer of security. Using our coffee shop analogy, now customers must not only have visited before but must order from the same counter they used previously. This is implemented by adding source port checking to our rules:

iptables -A INPUT -i $EXTIF -p tcp --sport $P --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $EXTIF -p udp --sport $P --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT

Notice the addition of --sport $P in our rules, ensuring both source and destination ports match.

4. Symmetric NAT: The Dynamic Assignment

Symmetric NAT is like having a different business card for each client you meet. Each connection gets its own unique mapping, making it the most secure but also potentially the most problematic for certain applications:

iptables -t nat -I POSTROUTING -s $INTERNAL_IP -o $EXTIF -j MASQUERADE

This simple rule creates dynamic mappings for each connection, ensuring maximum isolation.

Best Practices and Monitoring

To ensure your NAT implementation runs smoothly, consider these practices:

Enable connection tracking:

modprobe nf_conntrack
echo 1048576 > /proc/sys/net/netfilter/nf_conntrack_max

Monitor your NAT translations:

conntrack -L
iptables -t nat -L -v -n

Make your rules persistent:

netfilter-persistent save

Choosing the Right NAT Type

Your choice of NAT type should balance security with functionality:

Full Cone NAT offers the best compatibility but lowest security
Restricted Cone NAT provides a good balance for most applications
Port Restricted Cone NAT offers enhanced security while maintaining decent compatibility
Symmetric NAT provides maximum security but may break some applications

Conclusion

Understanding and implementing different NAT types is crucial for network security and functionality. Each type offers its own balance of security and accessibility, and choosing the right one depends on your specific needs. Remember to test thoroughly and monitor your implementation to ensure it meets your requirements.

Remember that NAT is just one part of a comprehensive network security strategy. Always combine it with proper firewalling and access controls to create a robust network infrastructure.

Have you implemented different types of NAT in your network? What challenges did you face? Share your experiences on the forum.

Understanding NAT Types Part 2 🔍

Tue, 14 Jan 2025 00:00:00 GMT

Let's start by breaking down each NAT type and implementing them properly with port specifications:

Understanding NAT Variables

First, let's define our variables clearly:

# Network interfaces
EXTIF="eth0"      # External/public interface
INTIF="eth1"      # Internal/private interface

# IP addresses
EXTERNAL_IP="192.168.2.170"  # Public IP address
INTERNAL_IP="10.0.0.1"       # Private IP address

# Port to be NAT-ed
P="12345"         # Example port number - replace with your desired port

Full Cone NAT Implementation

The key insight from the previous learning is that port matching is crucial. Here's the implementation:

# Clear existing NAT rules
iptables -t nat -F

# Outbound traffic translation (both TCP and UDP)
iptables -t nat -A POSTROUTING -o $EXTIF -p tcp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A POSTROUTING -o $EXTIF -p udp --sport $P -j SNAT --to-source $EXTERNAL_IP

# Inbound traffic translation (both TCP and UDP)
iptables -t nat -A PREROUTING -i $EXTIF -p tcp --dport $P -j DNAT --to-destination $INTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p udp --dport $P -j DNAT --to-destination $INTERNAL_IP

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

The key difference here is we're explicitly matching ports with --sport $P and --dport $P. This ensures that only traffic on the specified port is NAT-ed, maintaining the port-specific mapping that Full Cone NAT requires.

Restricted Cone NAT Implementation

Building on Full Cone NAT, we add connection state tracking:

# Base NAT rules (same as Full Cone)
iptables -t nat -A POSTROUTING -o $EXTIF -p tcp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A POSTROUTING -o $EXTIF -p udp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p tcp --dport $P -j DNAT --to-destination $INTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p udp --dport $P -j DNAT --to-destination $INTERNAL_IP

# Connection state tracking rules
iptables -A INPUT -i $EXTIF -p tcp --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $EXTIF -p udp --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT

# Drop new incoming connections unless they're related to existing ones
iptables -A INPUT -i $EXTIF -p tcp --dport $P -m state --state NEW -j DROP
iptables -A INPUT -i $EXTIF -p udp --dport $P -m state --state NEW -j DROP

Port Restricted Cone NAT Implementation

This builds on Restricted Cone NAT but adds source port matching:

# Base NAT rules (same as before)
iptables -t nat -A POSTROUTING -o $EXTIF -p tcp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A POSTROUTING -o $EXTIF -p udp --sport $P -j SNAT --to-source $EXTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p tcp --dport $P -j DNAT --to-destination $INTERNAL_IP
iptables -t nat -A PREROUTING -i $EXTIF -p udp --dport $P -j DNAT --to-destination $INTERNAL_IP

# Connection state tracking with port matching
iptables -A INPUT -i $EXTIF -p tcp --sport $P --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -i $EXTIF -p udp --sport $P --dport $P -m state --state ESTABLISHED,RELATED -j ACCEPT

# Drop new connections
iptables -A INPUT -i $EXTIF -p tcp --dport $P -m state --state NEW -j DROP
iptables -A INPUT -i $EXTIF -p udp --dport $P -m state --state NEW -j DROP

Symmetric NAT Implementation

For Symmetric NAT, we use MASQUERADE to create dynamic mappings:

# Clear existing rules
iptables -t nat -F

# Single rule for dynamic NAT mapping
iptables -t nat -I POSTROUTING -s $INTERNAL_IP -o $EXTIF -j MASQUERADE

To verify your NAT configuration is working as expected, you can use these helpful commands:

# View current NAT translations
conntrack -L

# Check iptables rules in NAT table
iptables -t nat -L -v -n

# Monitor NAT connections in real-time
watch -n 1 'conntrack -L | grep $EXTERNAL_IP'

Some important considerations:

Always enable connection tracking for all NAT types except Full Cone:

modprobe nf_conntrack
echo 1048576 > /proc/sys/net/netfilter/nf_conntrack_max

The rules assume your default policies are set appropriately. You might need to set them:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

Remember to save your rules to make them persistent:

# On Debian/Ubuntu:
netfilter-persistent save
# On CentOS/RHEL:
service iptables save

This implementation follows the correct port-specific behavior required by the NAT specifications while maintaining security through proper connection state tracking. Each type builds upon the previous one, adding more restrictions as we move from Full Cone to Symmetric NAT.

Mastering Agentless Monitoring with Zabbix on Cloudpanel 🔍

Mon, 06 Jan 2025 00:00:00 GMT

Understanding Agentless Monitoring Architecture

Modern infrastructure monitoring requires flexible, scalable solutions that can adapt to diverse environments. Zabbix's agentless monitoring represents a sophisticated approach to system observation, eliminating the need for agent deployment while maintaining robust monitoring capabilities.

Core Components and Protocols

Agentless monitoring leverages several key protocols for data collection:

SSH for secure command execution
SNMP for network device monitoring
IPMI for hardware health metrics
HTTP/HTTPS for web service monitoring

Implementation Guide

Configuring SSH Access

First, let's establish secure SSH access for Zabbix:

#!/bin/bash

# Configuration variables
ZABBIX_SERVER_KEY="/etc/zabbix/ssh_keys/id_rsa.pub"
TARGET_USER="zabbix-monitor"
TARGET_GROUP="zabbix-monitor"

# Create monitoring user with restricted privileges
sudo useradd -r -s /bin/bash -m -d /home/${TARGET_USER} ${TARGET_USER}
sudo mkdir -p /home/${TARGET_USER}/.ssh
sudo chmod 700 /home/${TARGET_USER}/.ssh

# Set up authorized keys with restricted commands
cat <<EOF > /home/${TARGET_USER}/.ssh/authorized_keys
command="uptime,free,mpstat" $(cat ${ZABBIX_SERVER_KEY})
EOF

sudo chmod 600 /home/${TARGET_USER}/.ssh/authorized_keys
sudo chown -R ${TARGET_USER}:${TARGET_GROUP} /home/${TARGET_USER}/.ssh

Basic Monitoring Template

Here's our enhanced monitoring script that includes CloudPanel-specific metrics:

#!/bin/bash

# System metrics collection
collect_system_metrics() {
    echo "=== System Metrics ==="
    uptime
    free -m
    mpstat
    
    # CloudPanel specific metrics
    if [ -d "/home/clp" ]; then
        echo "=== CloudPanel Metrics ==="
        # Nginx status
        systemctl status nginx | grep "Active:"
        # MySQL connections
        mysql -N -e "SHOW STATUS LIKE 'Threads_connected';"
        # PHP-FPM pools
        find /etc/php-fpm.d/ -name "*.conf" -exec basename {} \;
    fi
}

# Main execution
collect_system_metrics

Advanced Zabbix Templates

Let's create a sophisticated monitoring template that incorporates both standard system metrics and CloudPanel-specific monitoring:

// Template definition for Zabbix API
const cloudPanelTemplate = {
    template: "Template CloudPanel Advanced Monitoring",
    groups: {
        name: "CloudPanel Servers"
    },
    items: [
        {
            name: "Nginx Status",
            key_: "custom.nginx.status",
            type: 0,
            value_type: 3,
            delay: "1m"
        },
        {
            name: "MySQL Active Connections",
            key_: "custom.mysql.connections",
            type: 0,
            value_type: 3,
            delay: "2m"
        },
        {
            name: "PHP-FPM Pool Count",
            key_: "custom.phpfpm.pools",
            type: 0,
            value_type: 3,
            delay: "5m"
        }
    ],
    triggers: [
        {
            name: "Nginx Service Down",
            expression: "{Template CloudPanel Advanced Monitoring:custom.nginx.status.last()}=0",
            priority: 4
        },
        {
            name: "High MySQL Connections",
            expression: "{Template CloudPanel Advanced Monitoring:custom.mysql.connections.last()}>100",
            priority: 3
        }
    ]
};

CloudPanel Log Analysis and Traffic Monitoring

Before diving into advanced monitoring scenarios, let's explore CloudPanel's unique log structure and implement sophisticated traffic analysis. CloudPanel organizes logs by user, with each user having their own nginx access logs. This structure requires special consideration in our monitoring approach.

Implementing Traffic Analysis

The following script demonstrates how to monitor nginx traffic patterns and detect potential issues across all CloudPanel users:

#!/bin/bash

# Constants for configuration
ALERT_THRESHOLD_REQUESTS=500    # Threshold for high traffic from single IP
ERROR_THRESHOLD=100             # Threshold for error rate monitoring
SAMPLE_SIZE=1000               # Number of recent log entries to analyze
LOG_BASE_PATH="/home"          # Base path for user directories

analyze_nginx_traffic() {
    echo "Starting nginx traffic analysis..."
    
    # Iterate through each user's directory
    for user_dir in ${LOG_BASE_PATH}/*/; do
        username=$(basename "$user_dir")
        
        # Skip CloudPanel system user
        if [ "$username" != "clp" ] && [ -d "${user_dir}logs/nginx" ]; then
            # Process each access log file
            for access_log in "${user_dir}logs/nginx"/*access.log; do
                if [ -f "$access_log" ]; then
                    # Analyze high traffic patterns
                    local high_traffic_ips=$(tail -n ${SAMPLE_SIZE} "$access_log" | 
                        awk '{print $1}' | 
                        sort | 
                        uniq -c | 
                        sort -nr | 
                        head -n 5)
                    
                    # Get the highest traffic count
                    local top_ip_count=$(echo "$high_traffic_ips" | 
                        head -n 1 | 
                        awk '{print $1}')
                    
                    # Alert on high traffic
                    if [ "${top_ip_count:-0}" -gt ${ALERT_THRESHOLD_REQUESTS} ]; then
                        echo "High traffic detected for user: ${username}"
                        echo "Top IPs and request counts:"
                        echo "${high_traffic_ips}"
                    fi
                    
                    # Monitor for error rates
                    local error_count=$(tail -n ${SAMPLE_SIZE} "$access_log" | 
                        awk '$9 ~ /^[45]/ {print $9}' | 
                        wc -l)
                    
                    # Alert on high error rates
                    if [ "${error_count:-0}" -gt ${ERROR_THRESHOLD} ]; then
                        local error_details=$(tail -n ${SAMPLE_SIZE} "$access_log" | 
                            awk '$9 ~ /^[45]/ {print $1,$9,$7}' | 
                            tail -n 10)
                        echo "High error rate for user: ${username}"
                        echo "Recent errors:"
                        echo "${error_details}"
                    fi
                fi
            done
        fi
    done
}

# Execute the analysis
analyze_nginx_traffic

This script performs several important monitoring tasks:

Traverses user-specific log directories in CloudPanel's structure
Analyzes recent traffic patterns for potential DDoS or abuse
Monitors error rates to detect application issues
Generates detailed reports for investigation

Integrating with Zabbix

To incorporate this CloudPanel-specific monitoring into Zabbix, we can create custom items that process these logs:

#!/bin/bash

# Function to format data for Zabbix
format_zabbix_data() {
    local hostname="$1"
    local key="$2"
    local value="$3"
    echo "${hostname} ${key} ${value}"
}

# Process log data for Zabbix
process_logs_for_zabbix() {
    local hostname=$(hostname)
    
    for user_dir in /home/*/; do
        username=$(basename "$user_dir")
        if [ "$username" != "clp" ] && [ -d "${user_dir}logs/nginx" ]; then
            # Calculate metrics
            local total_requests=$(find "${user_dir}logs/nginx" -name "*access.log" -exec wc -l {} \; | awk '{sum+=$1} END {print sum}')
            local error_rate=$(find "${user_dir}logs/nginx" -name "*access.log" -exec awk '$9 ~ /^[45]/' {} \; | wc -l)
            
            # Format for Zabbix
            format_zabbix_data "$hostname" "cloudpanel.user[$username,requests]" "$total_requests"
            format_zabbix_data "$hostname" "cloudpanel.user[$username,errors]" "$error_rate"
        fi
    done
}

# Execute and send to Zabbix
process_logs_for_zabbix | zabbix_sender -c /etc/zabbix/zabbix_agentd.conf -i -

Creating Custom Triggers

Let's implement sophisticated triggers that account for CloudPanel's multi-user environment:

// Define triggers for CloudPanel user monitoring
const cloudPanelTriggers = {
    highTraffic: {
        name: "High Traffic Rate for {HOST.NAME} - User: {#USERNAME}",
        expression: "{CloudPanel:cloudpanel.user[{#USERNAME},requests].rate(5m)} > 1000",
        severity: "WARNING",
        description: "Detected high traffic rate for user {#USERNAME}"
    },
    errorSpike: {
        name: "Error Rate Spike for {HOST.NAME} - User: {#USERNAME}",
        expression: "{CloudPanel:cloudpanel.user[{#USERNAME},errors].min(5m)} > 100",
        severity: "HIGH",
        description: "High error rate detected for user {#USERNAME}"
    },
    sustainedLoad: {
        name: "Sustained High Load for {HOST.NAME} - User: {#USERNAME}",
        expression: "{CloudPanel:cloudpanel.user[{#USERNAME},requests].avg(15m)} > 500",
        severity: "AVERAGE",
        description: "Sustained high load detected for user {#USERNAME}"
    }
};

Advanced Monitoring Scenarios

CloudPanel Integration

When monitoring CloudPanel installations, we need to consider several unique aspects:

#!/bin/bash

# CloudPanel specific monitoring
NGINX_VHOST_DIR="/etc/nginx/sites-enabled"
PHP_FPM_DIR="/etc/php-fpm.d"
MYSQL_SOCK="/var/lib/mysql/mysql.sock"

# Monitor vhost configurations
monitor_vhosts() {
    local vhost_count=$(ls -1 ${NGINX_VHOST_DIR} | wc -l)
    local active_vhosts=$(nginx -T 2>/dev/null | grep "server_name" | wc -l)
    
    echo "Virtual Hosts: ${vhost_count}"
    echo "Active Configurations: ${active_vhosts}"
}

# Monitor PHP-FPM pools
monitor_php_pools() {
    for pool in ${PHP_FPM_DIR}/*.conf; do
        local pool_name=$(basename ${pool} .conf)
        local pool_status=$(systemctl is-active php-fpm@${pool_name})
        echo "Pool ${pool_name}: ${pool_status}"
    done
}

# Execute monitoring
monitor_vhosts
monitor_php_pools

Performance Optimization

For CloudPanel environments, we should monitor specific performance metrics:

# High load triggers for CloudPanel
{Template CloudPanel:system.cpu.util[,user].avg(5m)}>80 = High CPU Usage
{Template CloudPanel:vm.memory.util.avg(5m)}>90 = Critical Memory Usage
{Template CloudPanel:vfs.fs.size[/,pfree].last()}<10 = Low Disk Space Warning

# Service-specific triggers
{Template CloudPanel:net.tcp.service[http,,80].last()}=0 = Web Server Down
{Template CloudPanel:net.tcp.service[mysql].last()}=0 = Database Server Down

Best Practices and Considerations

Security Hardening

When implementing agentless monitoring, especially in CloudPanel environments, consider these security measures:

Use dedicated monitoring users with restricted permissions
Implement SSH key rotation
Set up IP-based access controls
Enable detailed audit logging

Scaling Considerations

For large-scale deployments:

Implement proxy servers for distributed monitoring
Use efficient polling intervals
Optimize trigger expressions
Configure appropriate history retention

Conclusion

Agentless monitoring with Zabbix provides a powerful solution for modern infrastructure monitoring. By incorporating CloudPanel-specific metrics and following best practices, you can build a robust monitoring system that scales with your needs while maintaining security and performance.

Remember to regularly review and update your monitoring templates as your infrastructure evolves, and always test new monitoring configurations in a staging environment before deploying to production.

Setting up Discord Notifications for OpenMediaVault 🔔

Tue, 23 Jan 2024 00:00:00 GMT

import CodeTabs from '$components/tabs/code-tabs.astro'

OpenMediaVault (OMV) is a powerful network-attached storage solution that typically relies on email for system notifications. However, many users prefer instant messaging platforms like Discord for monitoring their systems. This guide will walk you through setting up Discord notifications for OpenMediaVault, providing real-time system alerts directly in your Discord server.

Prerequisites

A running OpenMediaVault installation
Administrative access to your OMV system
A Discord server where you have permission to create webhooks
Basic familiarity with the Linux command line
SSH access to your OpenMediaVault system

Discord Webhook Setup

Open Discord and navigate to your server
Right-click on the desired notification channel
Select "Edit Channel" → "Integrations" → "Create Webhook"
Name your webhook (e.g., "OMV Notifications")
Copy the webhook URL for later use
Click "Save Changes"

Installation

Create Notification Script

Create a new file for the Discord notification script in OpenMediaVault:

nano /usr/share/openmediavault/notification/sink.d/20discord

</Fragment> </CodeTabs>

Add Script Content

Copy this script into the file, replacing the webhook URL placeholder:

#!/bin/bash
#CREATED BY XMON FOR OpenMediaVault
DISCORD_WEBHOOK='PUT YOUR WEBHOOK LINK HERE'

generate_post_data() {
  cat <<EOF
{
  "content": "",
  "embeds": [{
    "title": "${OMV_NOTIFICATION_SUBJECT}",
    "description": "$(cat ${OMV_NOTIFICATION_MESSAGE_FILE})",
    "color": "45973",
    "footer": {
        "text": "${OMV_NOTIFICATION_DATE}",
        "icon_url": ""
    }
  }]
}
EOF
}

curl -H "Content-Type: application/json" -X POST -d "$(generate_post_data)" $DISCORD_WEBHOOK

Make Script Executable

After saving the file, make it executable with:

chmod +x /usr/share/openmediavault/notification/sink.d/20discord

Configuration

Log into your OpenMediaVault web interface
Navigate to System → Notification
Keep email notifications enabled but disable authentication
Click "Save"
Test the setup using the "Test" button

Troubleshooting

If you encounter issues, here's a debugging version of the script that includes logging:

#!/bin/bash
DISCORD_WEBHOOK='YOUR_WEBHOOK_HERE'
LOG_FILE="/var/log/discord_notification.log"

# Log function
log_message() {
    echo "$(date): $1" >> $LOG_FILE
}

log_message "Script triggered"
log_message "Subject: ${OMV_NOTIFICATION_SUBJECT}"
log_message "Message file: ${OMV_NOTIFICATION_MESSAGE_FILE}"

# Check variables
if [ -z "$OMV_NOTIFICATION_SUBJECT" ]; then
    log_message "ERROR: Subject is empty"
fi

if [ ! -f "$OMV_NOTIFICATION_MESSAGE_FILE" ]; then
    log_message "ERROR: Message file does not exist"
fi

generate_post_data() {
  cat <<EOF
{
  "content": "",
  "embeds": [{
    "title": "${OMV_NOTIFICATION_SUBJECT}",
    "description": "$(cat ${OMV_NOTIFICATION_MESSAGE_FILE})",
    "color": "45973",
    "footer": {
        "text": "${OMV_NOTIFICATION_DATE}",
        "icon_url": ""
    }
  }]
}
EOF
}

# Log the curl attempt
log_message "Attempting to send to Discord"
RESPONSE=$(curl -H "Content-Type: application/json" -X POST -d "$(generate_post_data)" $DISCORD_WEBHOOK 2>&1)
log_message "Curl response: $RESPONSE"

Common Issues

File Permissions: Check permissions with:
```
ls -l /usr/share/openmediavault/notification/sink.d/20discord
```
Should show: -rwxr-xr-x 1 root root
System Logs: Review notification errors:
```
grep -i notification /var/log/syslog
```

Test Webhook: Verify webhook functionality:

curl -H "Content-Type: application/json" -X POST -d '{"content":"Test message"}' YOUR_WEBHOOK_URL

Security Notes

Keep your webhook URL private and secure
Monitor notification logs regularly
Consider implementing rate limiting for high-volume notifications
Update webhook URL if Discord server configurations change

Customization

The script can be customized by modifying:

Embed color (change the "color" value)
Footer icon (add an "icon_url")
Message format
Additional embed fields

Remember to check the debug log at /var/log/discord_notification.log if you make any modifications to ensure everything works as expected.

JavaScript/TypeScript Gameloop 🕹️

Tue, 14 May 2024 00:00:00 GMT

import CodeTabs from '$components/tabs/code-tabs.astro' import CodeEditor from '$components/code-editor/code-editor.svelte'

Overview

This snippet provides a game loop in JavaScript and TypeScript that runs every frame, ensuring more consistent values. It's a basic snippet that can be modified later to include additional functionalities, such as handling tab switching (as JavaScript is throttled when a tab is switched).

The Gameloop Snippet

<CodeTabs> <Fragment slot="JavaScript"> ```js wrap // To store last timestep let lastTime = 0; // To store time elapsed between frames let deltaTime = 0;

    // Function for the game loop that will be on loop forever.
    function gameLoop(currentTime) {
        // Calculate deltaTime by subtracting the last timestamp from the current timestamp and converting it to seconds
        deltaTime = (currentTime - lastTime) / 1000;
        
        // Update lastTime with the current timestamp
        lastTime = currentTime;

        // Call the update function to update game objects based on deltaTime
        update(deltaTime);

        // Call the render function to render game objects
        render();

        // Request the next frame by recursively calling gameLoop
        requestAnimationFrame(gameLoop);
    }

    // Function to update game logic
    function update(deltaTime) {
        // Placeholder function to update game objects based on deltaTime
        // Example: object.position.x += object.velocity.x * deltaTime;
        // Example 2: generators.map((generator) => generator.update(deltaTime))
    }

    // Function to render game objects
    function render() {
        // Placeholder function to render game objects. You don't need it if you are using frameworks.
        // Example: sprite.style.hidden = true or false.
    }

    // Start the game loop by requesting the first frame
    requestAnimationFrame(gameLoop);
    ```
</Fragment>
<Fragment slot="TypeScript">
    ```ts wrap
    // To store last timestep
    let lastTime: number = 0;
    // To store time elapsed between frames
    let deltaTime: number = 0;

    // Function for the game loop that will be on loop forever.
    function gameLoop(currentTime: number): void {
        // Calculate deltaTime by subtracting the last timestamp from the current timestamp and converting it to seconds
        deltaTime = (currentTime - lastTime) / 1000;
        
        // Update lastTime with the current timestamp
        lastTime = currentTime;

        // Call the update function to update game objects based on deltaTime
        update(deltaTime);

        // Call the render function to render game objects
        render();

        // Request the next frame by recursively calling gameLoop
        requestAnimationFrame(gameLoop);
    }

    // Function to update game logic
    function update(deltaTime: number): void {
        // Placeholder function to update game objects based on deltaTime
        // Example: object.position.x += object.velocity.x * deltaTime;
        // Example 2: generators.map((generator) => generator.update(deltaTime))
    }

    // Function to render game objects
    function render(): void {
        // Placeholder function to render game objects. You don't need it if you are using frameworks.
        // Example: sprite.style.hidden = true or false.
    }

    // Start the game loop by requesting the first frame
    requestAnimationFrame(gameLoop);
    ```
</Fragment>

</CodeTabs>

Code Editor

<CodeEditor title="Counter Increment" baseHtml={ <p> Click the button below to start the <span>Timer.</span> </p> <em>Check the JavaScript Tab to see the logic.</em> <button>Click to Start</button> } baseStyles={` /* Base Styles - you are free to edit them or yoink, it's only for PRESENTATION PURPOSE */

    body {
        font-family: system-ui;
        color: white;
        background: #262434;
        /* Centering */
        display: grid;
        justify-items: center;
        align-items: center;
    }

    /* Paragraph Styles */
    p {
        font-weight: bold;
        font-size: 1rem;
        text-align: center;
    }

    /* Gradient Button Styles */
    button {
        cursor: pointer;
        color: #c8a9ef;
        font-family: inherit;
        font-size: 1rem;
        border: 0;
        border-radius: 0.5rem;
        padding: 1rem;
        margin-top: 1rem;
        background: #8e2de2;
        background: linear-gradient(to bottom left, #4a00e0, #8e2de2);
        background-size: 200%;
        transition:
            background-size 250ms linear,
            color 250ms linear,
            transform 75ms linear;
    }

    button:hover {
        background-size: 100%;
        color: white;
        transform: scale(1.1);
    }
`}
baseScript={`
    // Variables that we will change and keep track of in countdown timer
    let isPlaying = false;
    let counter = 0;

    // References of DOM Elements
    const playButton = document.querySelector("button");
    const paragraph = document.querySelector("p");

    // Toggle true or false
    playButton.addEventListener("click", () => (isPlaying = !isPlaying));

    let lastTime = 0;
    let deltaTime = 0;

    function gameLoop(currentTime) {
        deltaTime = (currentTime - lastTime) / 1000;
        lastTime = currentTime;
        update(deltaTime);
        render();
        requestAnimationFrame(gameLoop);
    }

    function update(deltaTime) {
        if (isPlaying) {
            counter += 1;
            paragraph.innerHTML = counter;
        }
    }

    function render() {}
    // Start Loop
    requestAnimationFrame(gameLoop);
`}
client:idle />

Multi-Cloud Network Performance Monitoring Script 📊

Thu, 23 Jan 2025 00:00:00 GMT

import CodeTabs from '$components/tabs/code-tabs.astro'

Prerequisites

Linux/Unix environment with bash shell
SSH access to your cloud instances
iperf3 installed locally
Cloud instance with:
- Open ports for SSH (22), iperf3 (5201), and ICMP (ping)
- Public IP address
- SSH key-based authentication configured

Installation

First, ensure you have iperf3 installed on your local machine:

sudo apt-get update && sudo apt-get install -y iperf3

</Fragment> <Fragment slot="rhel">

sudo yum install -y iperf3

</Fragment> <Fragment slot="mac">

brew install iperf3

</Fragment> </CodeTabs>

Script Implementation

Create a new file named cloud-network-monitor.sh:

#!/bin/bash

# Cloud provider configurations
# Uncomment and configure your desired cloud provider

# AWS Configuration
#CLOUD_HOST="ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com"
#KEY_PATH="/root/.ssh/aws_key"

# Oracle Cloud Configuration
#CLOUD_HOST="ubuntu@xxx.xxx.xxx.xxx"  # Oracle instances use the public IP
#KEY_PATH="/root/.ssh/oracle_key"

# DigitalOcean Configuration
#CLOUD_HOST="root@xxx.xxx.xxx.xxx"    # DigitalOcean droplets typically use root user
#KEY_PATH="/root/.ssh/digitalocean_key"

# Common Configuration
HOME_VM="root@10.24.7.117"
INTERVAL=60
LOG_FILE="network_metrics.log"

# Function to validate configuration
validate_config() {
    if [ -z "$CLOUD_HOST" ] || [ -z "$KEY_PATH" ]; then
        echo "Error: Please configure CLOUD_HOST and KEY_PATH variables"
        exit 1
    fi
    
    if [ ! -f "$KEY_PATH" ]; then
        echo "Error: SSH key not found at $KEY_PATH"
        exit 1
    fi
}

# Function to check network metrics
check_network() {
    local cloud_ip=$(echo $CLOUD_HOST | cut -d@ -f2)
    echo "=== Network Test $(date) ===" | tee -a $LOG_FILE
    echo "Testing connection to: $CLOUD_HOST" | tee -a $LOG_FILE
    
    # Test ping latency (5 packets)
    echo "Testing latency..." | tee -a $LOG_FILE
    ping -c 5 $cloud_ip | tee -a $LOG_FILE
    
    # Test bandwidth using iperf3
    echo "Testing bandwidth..." | tee -a $LOG_FILE
    
    # Start iperf3 server on cloud VM with error handling
    if ! ssh -i $KEY_PATH -o StrictHostKeyChecking=accept-new $CLOUD_HOST "which iperf3 || sudo apt-get update && sudo apt-get install -y iperf3"; then
        echo "Failed to ensure iperf3 is installed on cloud server" | tee -a $LOG_FILE
        return 1
    fi
    
    if ! ssh -i $KEY_PATH $CLOUD_HOST "iperf3 -s -D"; then
        echo "Failed to start iperf3 server on cloud instance" | tee -a $LOG_FILE
        return 1
    fi
    
    # Wait for server to start
    sleep 2
    
    # Run bandwidth test locally
    iperf3 -c $cloud_ip -t 10 | tee -a $LOG_FILE
    
    # Stop iperf3 server on cloud VM
    ssh -i $KEY_PATH $CLOUD_HOST "pkill iperf3"
    
    echo "----------------------------------------" | tee -a $LOG_FILE
}

# Validate configuration before starting
validate_config

# Main loop
while true; do
    check_network || echo "Network check failed, will retry in $INTERVAL seconds"
    sleep $INTERVAL
done

Cloud Provider Setup

AWS Configuration

Configure your security group to allow:

Type: All ICMP - IPv4
Source: Your home IP

Type: Custom TCP
Port: 5201
Source: Your home IP

Type: SSH
Port: 22
Source: Your home IP

Ensure your SSH key is in OpenSSH format (if using .ppk):

puttygen /path/to/key.ppk -O private-openssh -o /root/.ssh/aws_key
chmod 600 /root/.ssh/aws_key

Oracle Cloud Configuration

Add the following security rules to your VCN:

TCP port 22 (SSH)
TCP port 5201 (iperf3)
ICMP Type 3 (ping)

DigitalOcean Configuration

Create firewall rules in the DigitalOcean console:

TCP port 22 (SSH)
TCP port 5201 (iperf3)
ICMP (ping)

Usage

Set proper permissions for the script:
```
chmod +x cloud-network-monitor.sh
```
Update the configuration section with your cloud instance details
Run the script:
```
./cloud-network-monitor.sh
```

The script will continuously monitor:

Network latency using ping
Bandwidth using iperf3
Results are logged to network_metrics.log

Troubleshooting

If you encounter SSH key issues:

# Fix key permissions
chmod 600 /path/to/your/key

# Test SSH connection
ssh -i /path/to/your/key -v your_cloud_host

For iperf3 connection issues:

# Test iperf3 manually
iperf3 -c your_cloud_ip -p 5201

Pangolin Stack Debugging Commands 📊

Tue, 28 Jan 2025 00:00:00 GMT

Pangolin Stack Debugging Commands

Container Management & Health

Basic Container Commands

# View all container stats (CPU, memory, network I/O)
docker stats

# View specific container stats
docker stats pangolin
docker stats gerbil
docker stats traefik

# Check container logs
docker compose logs -f
docker compose logs pangolin -f
docker compose logs gerbil -f
docker compose logs traefik -f

# Restart specific services
docker compose restart pangolin
docker compose restart gerbil
docker compose restart traefik

# Check container health status
docker inspect --format='{{.State.Health.Status}}' pangolin
docker inspect --format='{{.State.Health.Status}}' gerbil
docker inspect --format='{{.State.Health.Status}}' traefik

Container Inspection

# Get detailed container information
docker inspect pangolin
docker inspect gerbil
docker inspect traefik

# Check container environment variables
docker exec pangolin env
docker exec gerbil env
docker exec traefik env

# View running processes in containers
docker top pangolin
docker top gerbil
docker top traefik

Network Debugging

Network Inspection

# List all networks
docker network ls

# Inspect network details
docker network inspect pangolin_default

# Get container IP addresses
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' pangolin
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' gerbil
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' traefik

# Check network connectivity between containers
docker exec pangolin ping gerbil
docker exec gerbil ping traefik

Port Mapping

# List all port mappings
docker port pangolin
docker port gerbil
docker port traefik

# Check listening ports
docker exec pangolin netstat -tulpn
docker exec gerbil netstat -tulpn
docker exec traefik netstat -tulpn

Volume Management

Volume Inspection

# List all volumes
docker volume ls

# Inspect volume details
docker volume inspect pangolin_config
docker volume inspect traefik_config

# Check volume contents
docker exec pangolin ls -la /app/config
docker exec traefik ls -la /etc/traefik

Config File Access

# View configuration files
docker exec pangolin cat /app/config/config.yml
docker exec traefik cat /etc/traefik/traefik_config.yml
docker exec traefik cat /etc/traefik/dynamic_config.yml

# Check file permissions
docker exec pangolin ls -l /app/config
docker exec traefik ls -l /etc/traefik

Service-Specific Commands

Pangolin

# Check Pangolin API health
curl -I http://localhost:3001/api/v1/health

# View Pangolin logs
docker exec pangolin tail -f /app/config/logs/pangolin.log

# Check database status
docker exec pangolin ls -l /app/config/db

Gerbil

# Check WireGuard interface status
docker exec gerbil wg show
docker exec gerbil ip addr show

# View Gerbil peers
docker exec gerbil wg show all

# Check Gerbil logs
docker exec gerbil tail -f /var/log/gerbil.log

Traefik

# Check Traefik dashboard (if enabled)
curl -I http://localhost:8080

# View Traefik configurations
docker exec traefik cat /etc/traefik/traefik.yml

# Check SSL certificates
docker exec traefik ls -l /letsencrypt

# View access logs
docker exec traefik tail -f /var/log/access.log

Security & Certificate Management

SSL/TLS

# Check Let's Encrypt certificates
docker exec traefik cat /letsencrypt/acme.json

# View certificate expiration
docker exec traefik openssl x509 -in /letsencrypt/acme.json -text -noout

# Test SSL configuration
curl -vI https://your-domain.com

Authentication

# Check Badger plugin status
docker exec traefik curl http://localhost:8080/api/http/middlewares

# Test authentication endpoints
curl -I https://your-domain.com/api/v1/auth/status

CrowdSec Security Monitoring

Basic CrowdSec Operations

# Monitor CrowdSec container resources
docker stats crowdsec

# View CrowdSec metrics
curl http://localhost:6060/metrics | grep appsec
docker exec -it crowdsec cscli metrics

# Check CrowdSec logs
docker exec -it crowdsec tail -f /var/log/crowdsec.log
docker exec -it crowdsec ls -l /var/log/traefik/
docker exec -it crowdsec cscli explain /var/log/traefik/traefik.log

Decision Management

# List current decisions
docker exec -it crowdsec cscli decisions list

# Add captcha challenge for an IP
docker exec crowdsec cscli decisions add --ip 10.10.10.45 --type captcha -d 1h

# Ban an IP address
docker exec crowdsec cscli decisions add -i 10.10.10.45 -t ban -d 1h

Bouncer Management

# List all bouncers
docker exec crowdsec cscli bouncers list

# Add Traefik bouncer
docker exec crowdsec cscli bouncers add traefik-bouncer

# Remove Traefik bouncer
docker exec crowdsec cscli bouncers delete traefik-bouncer

Configuration and Integration

# View CrowdSec acquisition configuration
docker exec crowdsec cat /etc/crowdsec/acquis.yaml

# Restart CrowdSec and Traefik
docker compose restart traefik crowdsec

# Check CrowdSec network configuration
docker network inspect root_default | grep -A 5 pangolin

Performance Monitoring

Resource Usage

# Check container resource limits
docker inspect --format='{{.HostConfig.Resources}}' pangolin
docker inspect --format='{{.HostConfig.Resources}}' gerbil
docker inspect --format='{{.HostConfig.Resources}}' traefik

# Monitor real-time metrics
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

Network Performance

# Check network throughput
docker exec gerbil iftop -i wg0

# Monitor network connections
docker exec traefik netstat -ant | grep ESTABLISHED

Troubleshooting Tips

Always check logs first when encountering issues:

docker compose logs --tail=100 -f

Verify network connectivity between services:

docker network inspect pangolin_default

Check configuration file syntax:

docker exec pangolin yaml lint /app/config/config.yml
docker exec traefik traefik validate /etc/traefik/traefik_config.yml

Monitor system resources:

docker stats --no-stream

Verify service health endpoints:

curl -I http://localhost:3001/api/v1/health
curl -I http://localhost:3003/health # Gerbil
curl -I http://localhost:8080/ping # Traefik

Pangolin Database Commands 📊

Wed, 29 Jan 2025 00:00:00 GMT

Single-line SQLite commands that you can directly use in your terminal. Each command is designed for a specific administrative task:

Install sqlite3 to excuate commands

apt install sqlite3

Back up your SQLite database. This is an important practice for data safety. The command will create a backup file with the current date and time in its name.

Here's the command:

sqlite3 /path/to/db.sqlite ".backup '/path/to/backups/db_$(date +%Y%m%d_%H%M%S).sqlite'"

If you want to also compress the backup to save space, you can extend the command:

sqlite3 /path/to/db.sqlite ".backup '/path/to/backups/db_$(date +%Y%m%d_%H%M%S).sqlite'" && gzip "/path/to/backups/db_$(date +%Y%m%d_%H%M%S).sqlite"

This will create a compressed .gz file of your backup.

For User Management:

#### List all users and their roles
sqlite3 /path/to/db.sqlite "SELECT u.email, o.name as organization, r.name as role FROM user u LEFT JOIN userOrgs uo ON u.id = uo.userId LEFT JOIN orgs o ON uo.orgId = o.orgId LEFT JOIN roles r ON uo.roleId = r.roleId;"


#### Enable two-factor auth for a user
sqlite3 /path/to/db.sqlite "UPDATE user SET twoFactorEnabled = 1 WHERE email = 'user@email.com';"

For Site Management:

#### View all sites and their status
sqlite3 /path/to/db.sqlite "SELECT name, online, subnet FROM sites;"

#### Check site bandwidth usage
sqlite3 /path/to/db.sqlite "SELECT name, bytesIn/1024/1024 as MB_in, bytesOut/1024/1024 as MB_out FROM sites;"

For Resource Management:

#### List all resources and their domains
sqlite3 /path/to/db.sqlite "SELECT name, fullDomain, ssl FROM resources;"

#### Check resource access settings
sqlite3 /path/to/db.sqlite "SELECT name, blockAccess, sso, emailWhitelistEnabled FROM resources;"

For Role and Permission Management:

#### List all roles and their actions
sqlite3 /path/to/db.sqlite "SELECT r.name, a.actionId FROM roles r JOIN roleActions ra ON r.roleId = ra.roleId JOIN actions a ON ra.actionId = a.actionId;"

#### Show all admin users
sqlite3 /path/to/db.sqlite "SELECT u.email FROM user u JOIN userOrgs uo ON u.id = uo.userId WHERE uo.isOwner = 1;"

For Security Auditing:

#### List users with 2FA status
sqlite3 /path/to/db.sqlite "SELECT email, twoFactorEnabled, emailVerified FROM user;"

#### Show active sessions
sqlite3 /path/to/db.sqlite "SELECT s.id, u.email, s.expiresAt FROM session s JOIN user u ON s.userId = u.id;"

For Organization Management:

#### List all organizations and their domains
sqlite3 /path/to/db.sqlite "SELECT orgId, name, domain FROM orgs;"

#### Show organization members
sqlite3 /path/to/db.sqlite "SELECT o.name, u.email FROM orgs o JOIN userOrgs uo ON o.orgId = uo.orgId JOIN user u ON uo.userId = u.id;"

Each of these commands can be directly copied and pasted into your terminal. Remember to:

Replace /path/to/db.sqlite with your actual database path
Replace placeholder values like 'user@email.com' with actual values
Add quotes around values when dealing with strings

WinSCP with sudo privileges on an Ubuntu server AWS 📊

Tue, 28 Jan 2025 00:00:00 GMT

To run WinSCP with sudo privileges on an Ubuntu server, you can follow these detailed steps. This configuration allows you to access files and directories that require elevated permissions.

Prerequisites

Ensure you have WinSCP installed on your local machine.
You need to have access to an Ubuntu server (in this case, Ubuntu 24.04).
Your user account on the server must have the necessary permissions to use sudo.

Step-by-Step Guide

Open WinSCP:
- Launch the WinSCP application on your local machine.
Access Site Manager:
- Go to Session in the menu bar and select Sites followed by Site Manager.
Edit Your Site Profile:
- In the Site Manager window, select your existing site profile or create a new one.
- Click on the Edit button.
Open Advanced Settings:
- In the Edit Site dialog, click on the Advanced... button located at the bottom.
Configure SFTP Settings:
- In the Advanced Site Settings window, navigate to Environment and then select SFTP from the left panel.
- You will see a field labeled SFTP Server. Here, you need to enter the command that allows you to run the SFTP server with sudo privileges.
Insert SFTP Server Command:
- Enter the following command in the SFTP Server field:
```
sudo su -c /usr/lib/sftp-server
```
- Note: The path /usr/lib/sftp-server may vary based on your system configuration. You can verify the correct path by checking your SSH configuration file (/etc/ssh/sshd_config) for the line starting with Subsystem sftp.
Save Your Changes:
- Click OK to save your changes in the Advanced settings.
- Then click Save in the Edit Site dialog.
Connect to Your Server:
- Now, return to the main WinSCP interface and click on Login to connect to your server using your configured profile.
Verify Permissions:
- Once connected, you should have access to directories that require root permissions, allowing you to upload or modify files as needed.

Additional Considerations

Make sure that your user has NOPASSWD access in the /etc/sudoers file for seamless operation without being prompted for a password when using sudo commands. This can be set with an entry like:
```
username ALL=(ALL) NOPASSWD: ALL
```
Replace username with your actual username.
If you encounter issues connecting or transferring files, ensure that your SSH configuration allows for passwordless authentication if required, and check for any restrictions in your firewall settings or SSH daemon configuration.

By following these steps, you should be able to successfully configure WinSCP to operate with sudo privileges on your Ubuntu server, enhancing your file transfer capabilities while maintaining security protocols.

Installing Starship 🚀

Sat, 23 Mar 2024 00:00:00 GMT

import CodeTabs from '$components/tabs/code-tabs.astro'

Prerequisities

Terminal of your choice
Nerd Font - Any font with the Nerd Font patch to display icons in the terminal
Git Bash - I am using git bash primarily, you can use any other.

Installation

Install Starship using any package manager available on your machine. Detailed installation guides for various platforms are available on the official website.

Download using Package Manager

winget install -e --id Microsoft.WindowsTerminal

</Fragment> <Fragment slot="linux">

curl -sS https://starship.rs/install.sh | sh

</Fragment> <Fragment slot="mac">

curl -sS https://starship.rs/install.sh | sh

</Fragment> </CodeTabs>

Setup

Once installed, integrate Starship into your shell startup process:

Open your ~/.bashrc file (or equivalent for your shell).
Add the following line to the end of the file:
```
# ~/.bashrc
eval "$(starship init bash)"
```
Save the file and either restart your terminal or source the ~/.bashrc for the changes to take effect.