Creating Experiments

Learn how to create and manage A/B tests and feature flags with TokPulse.

Overview

Experiments in TokPulse allow you to:

  • Run A/B tests on your store
  • Implement feature flags
  • Test different configurations
  • Measure impact on key metrics

Experiment Types

A/B Tests

Traditional A/B tests compare two or more variants to determine which performs better.

Example: Testing different button colors

  • • Control: Blue button
  • • Treatment: Green button
  • • Metric: Click-through rate

Feature Flags

Feature flags allow you to enable/disable features for different user segments.

Example: New checkout flow

  • • Control: Current checkout
  • • Treatment: New checkout flow
  • • Rollout: Gradual (10% → 50% → 100%)

Creating an Experiment

1. Basic Setup

curl -X POST http://localhost:3001/api/experiments \
  -H "Content-Type: application/json" \
  -H "X-Org-ID: your-org-id" \
  -d '{
    "key": "checkout_flow_test",
    "name": "Checkout Flow Test",
    "description": "Test new checkout flow vs current",
    "allocation": 100,
    "variants": [
      {
        "key": "control",
        "name": "Current Flow",
        "weight": 50,
        "configJson": "{\"flow\": \"current\"}"
      },
      {
        "key": "treatment",
        "name": "New Flow",
        "weight": 50,
        "configJson": "{\"flow\": \"new\", \"steps\": 3}"
      }
    ]
  }'

Best Practices

Experiment Design

  1. Clear Hypothesis: Define what you're testing and why
  2. Single Variable: Test one change at a time
  3. Statistical Significance: Ensure adequate sample size
  4. Duration: Run for at least one business cycle

Configuration

  1. Fallbacks: Always provide fallback behavior
  2. Validation: Validate configuration JSON
  3. Testing: Test variants before launching
  4. Monitoring: Set up alerts and guardrails