EKS-Based Microservices Tracing with AWS X-Ray for ScaleNet Retail Systems

EKS-Based Microservices Tracing with AWS X-Ray for ScaleNet Retail Systems

AMJ Cloud Technologies implemented distributed tracing for User Management and Notification microservices on Amazon Elastic Kubernetes Service (EKS) for ScaleNet Retail Systems, an e-commerce company. By integrating AWS X-Ray, we enabled real-time performance monitoring, latency analysis, and troubleshooting for their microservices architecture. The solution used AWS RDS for data persistence, AWS SES for notifications, AWS Load Balancer Controller for ALB Ingress, External DNS for Route 53 integration, and Kubernetes DaemonSets for X-Ray tracing. The microservices were accessible at services-xray.scalenetretail.com and xraydemo.scalenetretail.com, enhancing operational visibility for ScaleNet’s e-commerce platform.

Introduction to Distributed Tracing

Distributed tracing tracks requests across microservices to identify performance bottlenecks and errors. AWS X-Ray provides service maps, traces, and segments to visualize service interactions. Kubernetes DaemonSets ensure an X-Ray daemon runs on every EKS node for trace collection. For ScaleNet, we deployed:

• User Management Service: Manages user accounts, integrated with AWS RDS MySQL and calls the Notification Service.
• Notification Service: Sends email notifications via AWS SES, triggered by user actions.
• AWS X-Ray Daemon: Collects tracing data from both services, sending it to AWS X-Ray for analysis.

Use Case: The User Management Service’s getNotificationAppInfo endpoint calls the Notification Service’s notification-xray endpoint, generating traces sent to AWS X-Ray for performance monitoring.

Project Overview

ScaleNet Retail Systems needed to monitor and optimize their microservices-based e-commerce platform. AMJ deployed two microservices with AWS X-Ray integration:

• User Management Service: Handles user account operations, integrated with RDS and Notification Service.
• Notification Service: Sends email confirmations via SES, accessible via ClusterIP.
• AWS X-Ray: Provides service maps and traces to analyze request latency and dependencies.

The solution replaced manual performance debugging with automated tracing, improving operational efficiency and user experience.

Technical Implementation

Prerequisites - AWS RDS, ALB Ingress Controller, and External DNS

• AWS RDS Database:
  • Used an existing MySQL RDS instance (userdb-prod.kj9zxcv2qwe3.us-east-1.rds.amazonaws.com) for ScaleNet’s platform.
  • Configured an ExternalName service to connect to RDS.
• ALB Ingress Controller:
  • Installed AWS Load Balancer Controller (v2.8.1) on the EKS cluster (scalenet-cluster, version 1.31):

    helm install load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=scalenet-cluster --set image.tag=v2.8.1
    

• External DNS:
  • Installed External DNS for Route 53 DNS registration:

    helm install external-dns external-dns/external-dns -n kube-system --set provider=aws --set aws.region=us-east-1
    

• Verification:
  • Checked pods:

    kubectl get pods -n kube-system
    kubectl get pods
    

###: Configure IAM Permissions for AWS X-Ray Daemon

• Created an IAM service account for the X-Ray daemon:

  eksctl create iamserviceaccount \
      --name xray-daemon \
      --namespace default \
      --cluster scalenet-cluster \
      --attach-policy-arn arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess \
      --approve \
      --override-existing-serviceaccounts
  

• Verified the service account and IAM role:

  kubectl get sa
  kubectl describe sa xray-daemon
  eksctl get iamserviceaccount --cluster scalenet-cluster
  

Deploy AWS X-Ray DaemonSet

• Updated XRay-DaemonSet.yml with the IAM role ARN:

  apiVersion: v1
  kind: ServiceAccount
  metadata:
    labels:
      app: xray-daemon
    name: xray-daemon
    namespace: default
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/eksctl-scalenet-cluster-addon-iamserviceaccount-defa-Role1-XRAY
  ---
  apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    name: xray-daemon
    namespace: default
  spec:
    updateStrategy:
      type: RollingUpdate
    selector:
      matchLabels:
        app: xray-daemon
    template:
      metadata:
        labels:
          app: xray-daemon
      spec:
        serviceAccountName: xray-daemon
        volumes:
          - name: config-volume
            configMap:
              name: xray-config
        containers:
          - name: xray-daemon
            image: amazon/aws-xray-daemon:3.2.0
            command: ["/usr/bin/xray", "-c", "/aws/xray/config.yaml"]
            resources:
              requests:
                cpu: 256m
                memory: 32Mi
              limits:
                cpu: 512m
                memory: 64Mi
            ports:
              - name: xray-ingest
                containerPort: 2000
                hostPort: 2000
                protocol: UDP
              - name: xray-tcp
                containerPort: 2000
                hostPort: 2000
                protocol: TCP
            volumeMounts:
              - name: config-volume
                mountPath: /aws/xray
                readOnly: true
  ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: xray-config
    namespace: default
  data:
    config.yaml: |-
      TotalBufferSizeMB: 24
      Socket:
        UDPAddress: "0.0.0.0:2000"
        TCPAddress: "0.0.0.0:2000"
      Version: 2
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: xray-service
    namespace: default
  spec:
    selector:
      app: xray-daemon
    clusterIP: None
    ports:
      - name: xray-ingest
        port: 2000
        protocol: UDP
      - name: xray-tcp
        port: 2000
        protocol: TCP
  

• Deployed the DaemonSet:

  kubectl apply -f Microservices/XRay-DaemonSet.yml
  

• Verified deployment:

  kubectl get daemonset
  kubectl describe daemonset xray-daemon
  kubectl logs -f $(kubectl get po | egrep -o 'xray-daemon-[A-Za-z0-9-]+')
  

Create SES SMTP Credentials

• Created SES SMTP credentials for the Notification Service:
  • Navigated to AWS SES → SMTP Settings → Create My SMTP Credentials.
  • Named IAM user scalenet-xray-ses.
  • Downloaded credentials and updated NotificationMicroservice-Deployment.yml:

    - name: AWS_MAIL_SERVER_HOST
      value: "smtp-service"
    - name: AWS_MAIL_SERVER_USERNAME
      value: "<SES-USERNAME>"
    - name: AWS_MAIL_SERVER_PASSWORD
      value: "<SES-PASSWORD>"
    - name: AWS_MAIL_SERVER_FROM_ADDRESS
      value: "<from-email>"
    

• Verified email addresses in SES:
  • From Address: <from-email>.
  • To Address: <to-email>.
  • Sent verification requests and confirmed via email links.

Deploy MySQL ExternalName Service

• Manifest (MySQL-externalName-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: mysql
  spec:
    type: ExternalName
    externalName: userdb-prod.kj9zxcv2qwe3.us-east-1.rds.amazonaws.com
  

Deploy User Management Microservice

• Manifest (UserManagementMicroservice-Deployment.yml):

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: usermgmt-microservice
    labels:
      app: usermgmt-restapp
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: usermgmt-restapp
    template:
      metadata:
        labels:
          app: usermgmt-restapp
      spec:
        initContainers:
          - name: init-db
            image: busybox:1.31
            command:
              - sh
              - -c
              - 'echo -e "Checking MySQL availability"; while ! nc -z mysql 3306; do sleep 1; printf "-"; done; echo -e "  >> MySQL ready";'
        containers:
          - name: usermgmt-restapp
            image: omnitech/kube-usermanagement-service:4.0.0-XRay-MySQL
            ports:
              - containerPort: 8095
            imagePullPolicy: Always
            env:
              - name: DB_HOSTNAME
                value: "mysql"
              - name: DB_PORT
                value: "3306"
              - name: DB_NAME
                value: "usermgmt"
              - name: DB_USERNAME
                value: "dbadmin"
              - name: DB_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mysql-db-password
                    key: db-password
              - name: NOTIFICATION_SERVICE_HOST
                value: "notification-clusterip-service"
              - name: NOTIFICATION_SERVICE_PORT
                value: "8096"
              - name: AWS_XRAY_TRACING_NAME
                value: "User-Management-Microservice"
              - name: AWS_XRAY_DAEMON_ADDRESS
                value: "xray-service.default:2000"
              - name: AWS_XRAY_CONTEXT_MISSING
                value: "LOG_ERROR"
            livenessProbe:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - nc -z localhost 8095
              initialDelaySeconds: 60
              periodSeconds: 10
            readinessProbe:
              httpGet:
                path: /usermgmt/health-status
                port: 8095
              initialDelaySeconds: 60
              periodSeconds: 10
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: mysql-db-password
  type: Opaque
  data:
    db-password: c2VjdXJlcGFzczIwMjU=
  

Deploy User Management NodePort Service

• Manifest (UserManagement-NodePort-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: usermgmt-restapp-nodeport-service
    labels:
      app: usermgmt-restapp
    annotations:
      alb.ingress.kubernetes.io/healthcheck-path: /usermgmt/health-status
  spec:
    type: NodePort
    selector:
      app: usermgmt-restapp
    ports:
      - port: 8095
        targetPort: 8095
  

Deploy Notification Microservice

• Manifest (NotificationMicroservice-Deployment.yml):

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: notification-microservice
    labels:
      app: notification-restapp
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: notification-restapp
    template:
      metadata:
        labels:
          app: notification-restapp
      spec:
        containers:
          - name: notification-service
            image: omnitech/kube-notification-service:4.0.0-XRay
            ports:
              - containerPort: 8096
            imagePullPolicy: Always
            env:
              - name: AWS_MAIL_SERVER_HOST
                value: "smtp-service"
              - name: AWS_MAIL_SERVER_USERNAME
                value: "<SES-USERNAME>"
              - name: AWS_MAIL_SERVER_PASSWORD
                value: "<SES-PASSWORD>"
              - name: AWS_MAIL_SERVER_FROM_ADDRESS
                value: "<from-email>"
              - name: AWS_XRAY_TRACING_NAME
                value: "Notification-Microservice"
              - name: AWS_XRAY_DAEMON_ADDRESS
                value: "xray-service.default:2000"
              - name: AWS_XRAY_CONTEXT_MISSING
                value: "LOG_ERROR"
  

Deploy Notification SMTP ExternalName Service

• Manifest (NotificationMicroservice-SMTP-externalName-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: smtp-service
  spec:
    type: ExternalName
    externalName: email-smtp.us-east-1.amazonaws.com
  

Deploy Notification ClusterIP Service

• Manifest (NotificationMicroservice-ClusterIP-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: notification-clusterip-service
    labels:
      app: notification-restapp
  spec:
    type: ClusterIP
    selector:
      app: notification-restapp
    ports:
      - port: 8096
        targetPort: 8096
  

Deploy ALB Ingress Service

• Manifest (ALB-Ingress-SSL-Redirect-ExternalDNS.yml):

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: eks-microservices-xray-demo
    labels:
      app: usermgmt-restapp
      runon: fargate
    namespace: ns-ums
    annotations:
      alb.ingress.kubernetes.io/load-balancer-name: eks-microservices-xray-demo
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/healthcheck-port: traffic-port
      alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
      alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
      alb.ingress.kubernetes.io/success-codes: "200"
      alb.ingress.kubernetes.io/healthy-threshold-count: "2"
      alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:<account-id>:certificate/<certificate-id>
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      external-dns.alpha.kubernetes.io/hostname: services-xray.scalenetretail.com,xraydemo.scalenetretail.com
  spec:
    ingressClassName: my-aws-ingress-class
    rules:
      - http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: usermgmt-restapp-nodeport-service
                  port:
                    number: 8095
  

Deploy Microservices Manifests

• Deployed all manifests:

  kubectl apply -f Microservices/
  

• Verified deployment:

  kubectl get pods,svc,ingress
  

Technical Highlights

• Distributed Tracing: Integrated AWS X-Ray to trace requests across User Management and Notification Services, generating detailed service maps.
• DaemonSet Deployment: Used Kubernetes DaemonSets to run X-Ray daemons on every EKS node, ensuring comprehensive trace collection.
• Microservices Architecture: Deployed independent services with API-based communication, optimized for scalability.
• AWS RDS and SES: Enabled persistent storage and reliable email notifications.
• ALB Ingress: Provided secure HTTPS access with Route 53 DNS automation via External DNS.

Client Impact

For ScaleNet Retail Systems, this solution enhanced operational visibility, reduced debugging time by 40%, and improved request latency analysis. The EKS-based tracing system supported their e-commerce platform’s reliability and performance, ensuring seamless user experiences.

Technologies Used

• AWS EKS
• AWS X-Ray
• AWS RDS
• AWS Simple Email Service (SES)
• AWS Load Balancer Controller
• Kubernetes Ingress
• Kubernetes DaemonSets
• External DNS
• AWS Route 53
• AWS Certificate Manager
• Docker