Skip to main content

Testing

This guide provides a complete set of tests to validate the infrastructure orchestration system end-to-end.

Table of Contents​

  1. Prerequisites
  2. Test 1: Full Initial Installation
  3. Test 2: Permissions Management
  4. Test 3: Updates and Restart
  5. Test 4: Clean and Reinstall
  6. Test 5: Troubleshooting Kafka
  7. Test 6: Services Validation
  8. Validation Checklist

Prerequisites​

Before running the tests, make sure you have:

# Verify Docker
docker --version
docker compose version

# Verify user permissions
groups | grep docker

# Verify the .env file exists
ls -la .env

# Verify the Docker network exists
source .env
docker network ls | grep "${DOCKER_NETWORK_NAME}"

If the network does not exist, create it:

source .env
docker network create "${DOCKER_NETWORK_NAME}"

Test 1: Full Initial Installation​

Goal​

Validate that the system installs correctly from scratch, respecting priorities and installation requirements.

Steps​

1.1 Full system cleanup​

# Stop all services
./infra.sh down all

# Remove all data (DESTRUCTIVE)
./infra.sh clean all --data

# Verify no containers are running
docker ps -a | grep -E "kafka|zookeeper|thingsboard|postgres"

Expected result: There should be no project-related containers.

1.2 Attempt to start services without install​

# This MUST fail because ThingsBoard requires installation
./infra.sh up all

Expected result:

Error: The following projects require installation before starting:
   - thingsboard

Please run the following commands first:
   ./infra.sh install thingsboard

1.3 Install ThingsBoard​

# Run installation
./infra.sh install thingsboard

# Verify the flag was created
ls -la platform/thingsboard/.installed

Expected result:

  • Installation completes without errors
  • .installed file exists
  • Database is initialized

1.4 Start all services​

# This should now work
./infra.sh up all

# Wait 30 seconds for services to stabilize
sleep 30

# Verify status
./infra.sh status all

Expected result:

  • All services are Up or healthy
  • Start order: Kafka (priority 30) → ThingsBoard (priority 50)

Test 2: Permissions Management​

Goal​

Validate that directory permissions are handled correctly, especially for Kafka/Zookeeper.

Steps​

2.1 Verify Kafka permissions​

# Stop Kafka
./infra.sh down kafka

# Check current permissions
ls -ln infrastructure/kafka/broker/data
ls -ln infrastructure/kafka/zookeeper/data
ls -ln infrastructure/kafka/zookeeper/logs

Expected result: Directories should be owned by 1000:1000

2.2 Simulate a permissions problem​

# Intentionally change ownership (requires sudo)
sudo chown -R root:root infrastructure/kafka/broker/data
sudo chown -R root:root infrastructure/kafka/zookeeper/data
sudo chown -R root:root infrastructure/kafka/zookeeper/logs

# Verify ownership is wrong
ls -ln infrastructure/kafka/broker/data

Expected result: Directories are now owned by 0:0 (root)

2.3 Start Kafka and verify auto-fix​

# The script should detect and fix permissions
./infra.sh up kafka

# Verify permissions were corrected
ls -ln infrastructure/kafka/broker/data
ls -ln infrastructure/kafka/zookeeper/data

Expected result:

  • The script prints permission-fix messages
  • Directories return to 1000:1000
  • Kafka starts correctly

2.4 Check Kafka logs​

# Verify there are no permission errors
docker logs kafka 2>&1 | grep -i "permission denied"
docker logs zookeeper 2>&1 | grep -i "permission denied"

Expected result: No permission errors should appear.

Test 3: Updates and Restart​

Goal​

Validate that updates and restarts work correctly without reinstalling.

Steps​

3.1 Restart individual services​

# Restart Kafka
./infra.sh restart kafka

# Wait 10 seconds
sleep 10

# Verify status
./infra.sh status kafka

Expected result: Kafka restarts without issues.

3.2 Restart ThingsBoard (without reinstall)​

# Restart ThingsBoard
./infra.sh restart thingsboard

# Wait 20 seconds
sleep 20

# Verify status
./infra.sh status thingsboard

Expected result:

  • ThingsBoard restarts correctly
  • It must NOT run the install process
  • Data must persist

3.3 ThingsBoard down + up​

# Stop ThingsBoard
./infra.sh down thingsboard

# Verify the .installed flag still exists
ls -la platform/thingsboard/.installed

# Start again
./infra.sh up thingsboard

# Wait 20 seconds
sleep 20

# Verify status
./infra.sh status thingsboard

Expected result:

  • .installed flag still exists
  • ThingsBoard starts normally (no install)
  • Data persists

3.4 Full stack update​

# Stop everything
./infra.sh down all

# Start everything (simulates an update)
./infra.sh up all

# Wait 30 seconds
sleep 30

# Verify status
./infra.sh status all

Expected result: Everything starts correctly without reinstalling.

Test 4: Clean and Reinstall​

Goal​

Validate the different cleanup levels and reinstall behavior.

Steps​

4.1 Clean without data (preserve data)​

# Clean without deleting data
./infra.sh clean thingsboard

# Verify data exists
ls -la platform/thingsboard/data

# Verify the .installed flag was NOT removed
ls -la platform/thingsboard/.installed

Expected result:

  • Containers are removed
  • Data persists
  • .installed flag persists

4.2 Clean with data (destructive)​

# Clean including data
./infra.sh clean thingsboard --data

# Verify data was removed
ls -la platform/thingsboard/data

# Verify the .installed flag was removed
ls -la platform/thingsboard/.installed

Expected result:

  • Containers are removed
  • Data is removed
  • .installed flag is removed

4.3 Reinstall after clean --data​

# Attempt to start without install (must fail)
./infra.sh up thingsboard

# Install again
./infra.sh install thingsboard

# Start
./infra.sh up thingsboard

# Verify status
./infra.sh status thingsboard

Expected result: The full reinstall cycle works.

Test 5: Troubleshooting Kafka​

Goal​

Diagnose and resolve common Kafka problems.

Steps​

5.1 Check Zookeeper connectivity​

# Connect to the Zookeeper container
docker exec -it zookeeper bash -c "echo stat | nc localhost 2181"

Expected result: Shows Zookeeper stats.

5.2 Check Kafka brokers​

# List brokers
docker exec -it kafka kafka-broker-api-versions --bootstrap-server localhost:9092

Expected result: Shows broker version information.

5.3 Check topics​

# List topics
docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 --list

Expected result: Shows existing topics (can be empty on a fresh install).

5.4 Create and test a topic​

# Create a test topic
docker exec -it kafka kafka-topics --bootstrap-server localhost:9092 \
  --create --topic test-topic --partitions 1 --replication-factor 1

# Produce a message
echo "test message" | docker exec -i kafka kafka-console-producer \
  --bootstrap-server localhost:9092 --topic test-topic

# Consume the message
docker exec -it kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 --topic test-topic --from-beginning --max-messages 1

Expected result: Topic is created and the message is produced and consumed successfully.

5.5 Check error logs​

# Kafka
docker logs kafka 2>&1 | tail -50

# Zookeeper
docker logs zookeeper 2>&1 | tail -50

Expected result: No critical errors.

Test 6: Services Validation​

Goal​

Validate that all services are working correctly and are reachable.

Steps​

6.1 Check exposed ports​

# List ports
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep -E "kafka|zookeeper|thingsboard"

Expected result:

  • Kafka: 9092, 9999
  • Zookeeper: 2181
  • ThingsBoard: 9090, 1883, 5683

6.2 Check HTTP connectivity​

# ThingsBoard Web UI
curl -I http://localhost:9090

# Kafka UI (if enabled)
curl -I http://localhost:8080

Expected result: HTTP 200 or 302 (redirect) responses.

6.3 Check health checks​

# Show health check status
docker ps --format "table {{.Names}}\t{{.Status}}" | grep -E "kafka|zookeeper|thingsboard"

Expected result: All show healthy or Up.

6.4 Check startup logs​

# ThingsBoard
docker logs thingsboard 2>&1 | grep -i "started"

# Kafka
docker logs kafka 2>&1 | grep -i "started"

Expected result: Successful startup messages.

Validation Checklist​

Use this checklist to validate that everything works correctly:

Initial Installation​

  • ./infra.sh up all fails if ThingsBoard is not installed
  • ./infra.sh install thingsboard completes successfully
  • .installed file is created
  • ./infra.sh up all works after installation
  • Services start in the correct order (priorities)

Permissions​

  • Kafka directories are 1000:1000
  • Zookeeper directories are 1000:1000
  • Kafka scripts detect and fix wrong permissions
  • No permission errors in logs

Updates​

  • ./infra.sh restart thingsboard works without reinstall
  • ./infra.sh down + ./infra.sh up works without reinstall
  • .installed flag persists after down
  • Data persists after restart

Clean​

  • ./infra.sh clean thingsboard preserves data and .installed
  • ./infra.sh clean thingsboard --data deletes data and .installed
  • After clean --data, install is required again

Kafka​

  • Zookeeper responds on port 2181
  • Kafka responds on port 9092
  • Topics can be created
  • Messages can be produced and consumed
  • No errors in logs

ThingsBoard​

  • Web UI reachable at http://localhost:9090
  • MQTT reachable on port 1883
  • Database initialized correctly
  • No errors in logs

Integration​

  • ThingsBoard can connect to Kafka
  • Microservices can connect to Kafka
  • All services are on the ${DOCKER_NETWORK_NAME} network

Common Problems and Fixes​

Kafka does not start (permissions error)​

Symptom:

kafka | [error] Permission denied: /var/lib/kafka/data

Fix:

./infra.sh down kafka
sudo chown -R 1000:1000 infrastructure/kafka/broker/data
sudo chown -R 1000:1000 infrastructure/kafka/zookeeper/data
sudo chown -R 1000:1000 infrastructure/kafka/zookeeper/logs
./infra.sh up kafka

ThingsBoard requires install after an update​

Symptom:

Error: thingsboard requires installation before starting.

Fix:

# Check whether the flag exists
ls -la platform/thingsboard/.installed

# If missing, reinstall
./infra.sh install thingsboard

Kafka cannot connect to Zookeeper​

Symptom:

kafka | [error] Connection to zookeeper:2181 failed

Fix:

# Verify Zookeeper is running
docker ps | grep zookeeper

# Check Zookeeper logs
docker logs zookeeper

# Restart in order
./infra.sh down kafka
./infra.sh up kafka

Docker network does not exist​

Symptom:

Error: network not found

Fix:

source .env
docker network create "${DOCKER_NETWORK_NAME}"

Automated Test Script​

You can run this script to perform a full test:

#!/bin/bash

echo "Starting full infrastructure test..."

# Test 1: Full clean
echo "Cleaning system..."
./infra.sh clean all --data

# Test 2: Install
echo "Installing ThingsBoard..."
./infra.sh install thingsboard

# Test 3: Start services
echo "Starting all services..."
./infra.sh up all
sleep 30

# Test 4: Verify status
echo "Checking status..."
./infra.sh status all

# Test 5: Verify Kafka
echo "Checking Kafka..."
docker exec kafka kafka-broker-api-versions --bootstrap-server localhost:9092

# Test 6: Verify ThingsBoard
echo "Checking ThingsBoard..."
curl -I http://localhost:9090

echo "Full test completed"

Important Notes​

  1. Priority order: Services start according to their priority (lower number = higher priority)

    • Kafka: 30
    • ThingsBoard: 50

  2. Installation flag: Only ThingsBoard requires the .installed flag

  3. Kafka permissions: Must always be 1000:1000

  4. Clean vs Clean --data:

    • clean: Removes containers, preserves data
    • clean --data: Removes containers AND data (destructive)

  5. Docker network: All services must be on the ${DOCKER_NETWORK_NAME} network