AWS Data Firehose Integration

​

Why AWS Data Firehose?

• Edge-level tracking — Captures logs at CloudFront edge locations
• Real-time delivery — Logs stream to ChatFeatured in real-time
• Built for scale — Handles high-volume traffic efficiently
• Enterprise-grade — Designed for large organizations
• Cost-effective — Pay per GB of logs processed

​

Prerequisites

• AWS Account with CloudFront and IAM permissions
• Active CloudFront distribution with content
• Python 3.12 knowledge (for Lambda understanding)
• Your ChatFeatured API Key (starts with aa_)
• S3 bucket access (for log backup)

​

Architecture Overview

CloudFront (Content Delivery)
    ↓
CloudFront Real-time Logs
    ↓
Data Firehose (Streaming Service)
    ↓
Lambda (Transform Logs)
    ↓
ChatFeatured Analytics
    ↓
S3 (Backup for failures)

​

Step-by-Step Setup

​

Step 1: Create Lambda Transformation Function

1. Sign in to AWS Lambda Console
2. Click Create function
3. Select Author from scratch
4. Configure:
   • Function name: chatfeatured-cloudfront-log-processor
   • Runtime: Python 3.12
   • Architecture: x86_64
5. Click Create function

import json
import base64
import urllib.request
import urllib.error
import os

def lambda_handler(event, context):
    """Transform CloudFront logs to ChatFeatured format"""
    
    output = []
    
    for record in event['records']:
        # Decode the log data
        payload = base64.b64decode(record['data']).decode('utf-8')
        
        # Parse CloudFront log line
        fields = payload.strip().split('\t')
        
        if len(fields) < 12:
            continue
        
        try:
            # Extract CloudFront fields
            log_entry = {
                'timestamp': f"{fields[0]}T{fields[1]}Z",  # date + time
                'host': fields[4],  # cs(Host)
                'method': fields[5],  # cs-method
                'pathname': fields[7],  # cs-uri-stem
                'query_params': fields[11] if len(fields) > 11 else '',
                'status': int(fields[8]),  # sc-status
                'ip': fields[2],  # c-ip
                'userAgent': fields[10] if len(fields) > 10 else '',
                'referer': fields[9] if len(fields) > 9 else '',
                'bytes_sent': int(fields[3]) if len(fields) > 3 else 0,
            }
            
            # Only send to ChatFeatured if it's an HTML page (200-299 status)
            if 200 <= log_entry['status'] < 300:
                send_to_chatfeatured(log_entry)
            
            # Prepare output for Firehose
            output_record = {
                'recordId': record['recordId'],
                'result': 'Ok',
                'data': base64.b64encode(
                    (json.dumps(log_entry) + '\n').encode('utf-8')
                ).decode('utf-8')
            }
        except Exception as e:
            print(f"Error processing record: {e}")
            output_record = {
                'recordId': record['recordId'],
                'result': 'ProcessingFailed',
            }
        
        output.append(output_record)
    
    return {'records': output}

def send_to_chatfeatured(log_entry):
    """Send log directly to ChatFeatured"""
    
    api_key = os.environ.get('CHATFEATURED_API_KEY')
    endpoint = 'https://ingest.chatfeatured.com/v1/logs/aws_data_firehose_cloudfront'
    
    try:
        req = urllib.request.Request(
            endpoint,
            data=json.dumps(log_entry).encode('utf-8'),
            headers={
                'Content-Type': 'application/json',
                'X-API-Key': api_key,
            }
        )
        urllib.request.urlopen(req, timeout=5)
    except urllib.error.URLError as e:
        print(f"Failed to send log: {e}")

6. Click Deploy
7. Note your Lambda function ARN (shown at top-right)

​

Step 2: Add Environment Variable to Lambda

1. In Lambda settings, go to Configuration → Environment variables
2. Click Edit
3. Add:
   • Key: CHATFEATURED_API_KEY
   • Value: Your ChatFeatured API Key (starts with aa_)
4. Click Save

​

Step 3: Create S3 Backup Bucket

1. Open S3 Console
2. Click Create bucket
3. Bucket name: chatfeatured-cloudfront-logs-backup-[your-account-id]
4. Region: Same as your CloudFront distribution
5. Click Create bucket

​

Step 4: Create Data Firehose Delivery Stream

1. Go to Amazon Data Firehose Console
2. Click Create Firehose stream
3. Configure source and destination:
   • Source: Direct PUT
   • Destination: HTTP Endpoint
4. Stream name: chatfeatured-cloudfront-analytics
5. Processing section:
   • Enable: “Transform source records” ✓
   • Select your Lambda from Step 1
6. HTTP endpoint destination:
   • URL: https://ingest.chatfeatured.com/v1/logs/aws_data_firehose_cloudfront
   • Access key: Your ChatFeatured API Key
   • Enable: GZIP compression ✓
7. S3 backup:
   • Backup mode: “Failed data only”
   • Select bucket from Step 3
8. Click Create Firehose stream

​

Step 5: Configure CloudFront Real-time Logs

1. Go to CloudFront Console
2. Select your distribution
3. Click Monitoring → Logs → Real-time logs
4. Click Create configuration
5. Configure:
   • Name: chatfeatured-agent-analytics
   • Sampling rate: 100 (capture all requests)
   • Log fields: Select these:
     • date, time, c-ip, cs-method, cs(Host), cs-uri-stem
     • cs-uri-query, cs(User-Agent), cs(Referer), sc-status
     • sc-bytes, time-taken
   • Log recipients: Select your Firehose stream from Step 4
   • Cache behaviors: “All cache behaviors”
6. Click Create configuration

​

Verifying the Setup

​

Check Data Flow

1. Generate traffic to your CloudFront distribution (visit your site)
2. Wait 2-3 minutes for logs to stream
3. Go to ChatFeatured Analytics Dashboard
4. You should see incoming traffic

​

Monitor CloudFront

1. In CloudFront Monitoring tab, check Real-time logs status
2. Should show Successful deliveries counter incrementing

​

Check Lambda Logs

1. In Lambda Monitor tab, check Recent invocations
2. Click invocation to see logs
3. Verify no errors in execution

​

Check Firehose

1. In Data Firehose console, select your stream
2. Check Delivery stream metrics:
   • Incoming records should be > 0
   • Delivered bytes should be increasing

​

Advanced Configuration

​

Filtering by CloudFront Behavior

1. In CloudFront, create cache behaviors for different path patterns
2. Add separate Real-time log configurations per behavior
3. Route to different Firehose streams if desired

​

Cost Optimization

• In CloudFront Real-time logs, reduce sampling rate to 10%
• Reduces cost by 90% but loses granularity

• Modify Lambda to filter certain paths (skip /health, /metrics)
• Keeps all traffic but filters noise

​

Lambda Error Handling

# Only logs successful responses (200-299)
if 200 <= log_entry['status'] < 300:
    send_to_chatfeatured(log_entry)

# Track any errors (400+)
if 400 <= log_entry['status'] < 600:
    log_entry['type'] = 'error'
    send_to_chatfeatured(log_entry)

​

Troubleshooting

​

Cost Estimation

​

Performance Notes

• Latency: Minimal, handled asynchronously by Firehose
• Log delay: 1-3 minute delay from request to ChatFeatured
• Throughput: Can handle millions of requests/hour
• Reliability: Automatic retry with S3 backup

​

Security Best Practices

• Store API Key in Lambda environment variables (encrypted at rest)
• Use IAM roles with minimal permissions
• Enable S3 encryption for backup bucket
• Monitor CloudFront logs for suspicious activity

​

See Also

• Cloudflare Workers — Simpler Cloudflare setup
• Vercel Log Drains — For Vercel hosting