The Core Problem: Workflows Change; Code Shouldn't

Building a customer onboarding system for PayPal Balance accounts sounds straightforward until you realize how many stakeholders need to touch the workflow:

·       Compliance teams need to add KYC verification steps

·       Legal teams need to update agreement flows based on regulations

·       Risk teams want to modify the sequence based on user segments

·       Product teams want to A/B test different checkpoint orders to optimize conversion

·       Account provisioning needs conditional flows based on account type

In a traditional approach, each workflow change means code changes. Each code change means a PR, code review, testing, and a deployment cycle. In a regulated financial environment, this becomes a bottleneck. You end up with engineers becoming gatekeepers for business decisions instead of enablers.

We needed a system where workflow definitions were configuration, not code.

Why JSON-Based Workflows Make Sense

Before diving into implementation, let's be clear about the scope. Our onboarding workflow has roughly 11-12 checkpoints that can be arranged in different sequences:

·       Contact verification (email OTP, phone OTP)

·       KYC verification (document scan, personal info, address info)

·       Risk checks (AML, source of income)

·       Legal agreements (consent, document signing)

The individual checkpoint logic is stable, we call specific downstream services for each one. But the sequence and combinations change frequently based on business requirements.

This is the key insight: if the building blocks are stable but the arrangements are variable, use configuration.

We designed our workflow definitions as JSON because:

1. Easy to edit and version control - JSON diffs clearly show what changed

2. Schema-based validation - We can validate structure before execution

3. Human-readable - Non-engineers can understand and review changes

4. Composable - Same checkpoints can be arranged into different workflows

Here's what an actual workflow definition looks like:

Each checkpoint is self-contained: it knows what data it requires, what the data types are, what fields are optional, and what constraints apply.

Real Use Cases We Designed For

During the design phase, we mapped out the actual scenarios we'd encounter:

Use Case 1: Regional Flow Variants

 Multiple workflow definitions.

Same endpoints, different workflow configurations. The service loads the right one based on the request headers (entity, type, category).

Use Case 2: Adding/Removing Fields Without Code Changes

A requirement comes to add “maternal_last_name” to KYC_INFO:

1. Add field to the KYC_INFO checkpoint schema in the JSON

2. Validate that incoming requests include this field

3. Done

4. We can keep the required: false and wait for upstream to integrate so nothing breaks even if we push the code and later once the system is stable change it to required: true

No code changes. No deployment. Just a configuration update.

Use Case 3: Conditional Fields

Sometimes a field should only be required if another field has a certain value. Instead of hardcoding this in Java, we express it in the schema:

This field is optional. The checkpoint handler can decide whether to require it based on the actual data received. The schema defines possibilities; the handler defines logic.

Use Case 4: Agreement Versioning

Legal needs to update the agreement template. They can't just overwrite it—regulatory audits require tracking which version each customer accepted. We handle this with version fields:

When a customer reaches the legal agreement checkpoint, they see the current version. We store which version they accepted.

Architecture: How It All Works Together

The system has three main layers:

1. Workflow Configuration Layer (JSON Files)

These are the source of truth. They define checkpoints, sequences, and data schemas. They're versioned in Git like any configuration.

2. Validation Layer (Schema-Based)

When a request comes in for a checkpoint, the validator:

·       Loads the workflow definition

·       Finds the specific checkpoint

·       Validates the request against that checkpoint's schema

·       Returns detailed errors if validation fails

This happens synchronously for every request.

3. Execution Layer (Checkpoint Handlers)

Once validated, the request goes to a handler. Each checkpoint type has a handler that knows how to call downstream services:

The handlers are stable. We add new handlers rarely. Most work is configuration changes.

Here's how the validation works in practice:

And the flow schema validation ensures the workflow itself is valid:

Key Design Decisions

Decision 1: Two-Layer Validation

We validate at two points:

·       Flow-level validation: Is this workflow combination supported?

·       Checkpoint-level validation: Does this request match the checkpoint's schema?

This catches configuration errors early and prevents bad data from reaching handlers.

Decision 2: Checkpoint Discovery by Naming Convention

Checkpoints are discovered by name, not registration:

This reduces boilerplate. Add a checkpoint to the JSON, and if a matching handler exists, it works. If not, we fail with a clear error.

Decision 3: Immutable Workflow Versions

When we start an onboarding transaction, we load and store the workflow version it uses. If we update the JSON later, existing transactions continue with their original workflow.

This prevents a customer halfway through the process from suddenly hitting new fields or reordered checkpoints.

Decision 4: Duplicate Data processing

In distributed systems, requests get retried. If a checkpoint execution succeeds but the response is lost, the customer might resubmit. We handle this with by pulling the data from onboarding data.

This prevents duplicate processing. If a customer verifies their email twice, the second request returns the same response as the first without calling the verification service again.

Decision 5: Factory Pattern for Handlers

We use a factory to instantiate the correct handler:

This allows:

·       Adding new handlers without modifying core logic

·       Testing handlers in isolation

·       Swapping implementations based on configuration

Implementation:

1. Schema-Based Validation

By expressing constraints in JSON instead of code, we catch configuration errors immediately. If someone updates the schema to require a new field, the next request that hits that checkpoint fails validation with a clear error message, long before the handler tries to process it.

2. Multi-cloud Support

We use Maven profiles to support AWS (DynamoDB), GCP (Spanner), and local deployments:

The workflow engine is cloud-agnostic. Different backends, same logic.

3. Comprehensive Logging

Every checkpoint execution is logged:

·       Input data

·       Validation results

·       Handler response

·       Execution time

·       Errors

This audit trail is essential. When something goes wrong, you need to know exactly what was received and what was returned.

4. Version Tracking for Agreements

We track major/minor versions for legal agreements. If the legal team updates an agreement template, new customers get the new version. Old customers continue with the version they accepted.

The Constraints Worth Knowing

Best fit when:

·       Workflows are business-driven (not engineering-driven)

·       The set of possible checkpoints is relatively stable

·       You have many workflow variants

·       Non-engineers need to modify workflows

·       Change velocity is high

Lessons Learned

1. Documentation is mandatory.

When workflow logic lives in JSON, your documentation is your source of truth. We maintain:

·       Checkpoint specifications (what each checkpoint does, what services it calls)

·       Schema documentation (what fields exist, when they're required)

·       Workflow diagrams (visual representation of checkpoint sequences)

·       Examples of common configurations

2. Validation needs to be comprehensive.

Because you can't rely on compile-time checking, runtime validation has to be solid. We validate:

·       Workflow structure (all required fields present)

·       Checkpoint definitions (valid checkpoint names, valid handler mappings)

·       Request data (matches the checkpoint schema)

·       State transitions (valid progression through checkpoints)

3. Version everything.

Workflows change. Schemas evolve. Checkpoint handlers get updated. Without versioning:

·       You can't rollback

·       You can't debug historical transactions

·       You can't support multiple variants

·       You can't do safe migrations

We version workflows with the file name and schemas within the JSON.

Conclusion:

This architecture is built on a simple idea: if it changes frequently and is decided by non-engineers, it should be configuration, not code.

Workflows fit that description perfectly. KYC requirements, legal agreements, compliance checks—these are business decisions. They shouldn't live in Java code.

By extracting workflows into JSON:

·       We gave decision-making power to the experts (compliance, legal, product)

·       We removed engineering as a bottleneck for business iteration

·       We enabled rapid testing and rollback of new flows

·       We created a clear audit trail of who changed what and when

The initial investment was real, we had to build validation, handler factories, versioning, and comprehensive logging. But the payoff is a system that moves at the speed of business, not the speed of engineering sprints.

Key Takeaways:

1. Extract workflows to configuration when business drives changes, not engineering

2. Build comprehensive schema validation to compensate for lack of compile-time checking

3. Use checkpoint handlers as stable building blocks; treat them like public APIs

4. Version everything like workflows, schemas, transactions

5. Invest heavily in logging, monitoring, and documentation

6. Idempotency keys are essential for safe retries in distributed systems

7. Two-layer validation catches errors early (flow level + checkpoint level)

For Your Team:

·       Identify workflows that change frequently

·       Map your discrete steps (checkpoints)

·       Design a schema that expresses your variations

·       Build validators and handler factories

·       Start with one workflow type

·       Add comprehensive observability before going to production

·       Document thoroughly