Week 5: Backend & Data Patterns

APIs and Data Architecture

Reading Assignment

Complete this reading before Week 6. Estimated time: 55-70 minutes.

Introduction

The previous weeks covered application architecture and frontend patterns. This week, we focus on the backend and data layers—the foundation that powers your applications. We’ll explore RESTful API design in depth, middleware patterns for cross-cutting concerns, and data architecture strategies including the concept of polyglot persistence.

Understanding these patterns is crucial because the backend is where business logic lives, where data is persisted, and where security is enforced. Poor backend design leads to applications that are slow, insecure, and difficult to maintain.

RESTful API Design

What is REST?

REST (Representational State Transfer) is an architectural style for designing networked applications. RESTful APIs use HTTP as the communication protocol and organize operations around resources—the nouns of your domain.

Key REST principles:

Client-Server Separation - Client and server evolve independently
Statelessness - Each request contains all information needed to process it
Uniform Interface - Consistent conventions for interacting with resources
Layered System - Clients can’t tell if they’re connected directly to the server
Cacheability - Responses indicate whether they can be cached

Resource-Oriented Design

Resources are the core concept in REST. Every URL represents a resource:

/api/users              → Collection of users
/api/users/123          → Specific user
/api/users/123/tasks    → Tasks belonging to user 123
/api/tasks/456          → Specific task

Guidelines for Resource URLs:

Guideline	Good	Avoid
Use nouns, not verbs	`/api/tasks`	`/api/getTasks`
Use plural nouns	`/api/users`	`/api/user`
Use lowercase	`/api/tasks`	`/api/Tasks`
Use hyphens for readability	`/api/task-categories`	`/api/taskCategories`
Nest for relationships	`/api/projects/123/tasks`	`/api/projectTasks?projectId=123`

HTTP Methods and CRUD Operations

HTTP methods map to CRUD operations:

Method	Operation	Endpoint	Purpose
GET	Read	`/api/tasks`	List all tasks
GET	Read	`/api/tasks/123`	Get task 123
POST	Create	`/api/tasks`	Create new task
PUT	Replace	`/api/tasks/123`	Replace task 123 entirely
PATCH	Update	`/api/tasks/123`	Update specific fields of task 123
DELETE	Delete	`/api/tasks/123`	Delete task 123

Method Properties:

Method	Idempotent	Safe	Request Body
GET	Yes	Yes	No
POST	No	No	Yes
PUT	Yes	No	Yes
PATCH	No	No	Yes
DELETE	Yes	No	Optional

Idempotent: Multiple identical requests produce the same result
Safe: Request doesn’t modify server state

HTTP Status Codes

Status codes communicate the result of a request:

2xx - Success

200 OK              - Request succeeded
201 Created         - Resource created (include Location header)
204 No Content      - Success with no response body (DELETE)

3xx - Redirection

301 Moved Permanently - Resource has new URL
304 Not Modified      - Cached version is still valid

4xx - Client Errors

400 Bad Request     - Invalid request syntax/data
401 Unauthorized    - Authentication required
403 Forbidden       - Authenticated but not authorized
404 Not Found       - Resource doesn't exist
409 Conflict        - Request conflicts with current state
422 Unprocessable   - Valid syntax but semantic errors

5xx - Server Errors

500 Internal Error  - Generic server error
502 Bad Gateway     - Upstream server error
503 Service Unavailable - Server temporarily unavailable

Request and Response Design

Consistent Response Structure:

// Success response
{
  "data": {
    "id": "123",
    "title": "Complete API design",
    "status": "pending"
  },
  "meta": {
    "timestamp": "2026-01-20T10:00:00Z"
  }
}

// Collection response with pagination
{
  "data": [
    { "id": "1", "title": "Task 1" },
    { "id": "2", "title": "Task 2" }
  ],
  "meta": {
    "total": 150,
    "page": 1,
    "perPage": 20,
    "totalPages": 8
  },
  "links": {
    "self": "/api/tasks?page=1",
    "next": "/api/tasks?page=2",
    "last": "/api/tasks?page=8"
  }
}

// Error response
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Request validation failed",
    "details": [
      { "field": "title", "message": "Title is required" },
      { "field": "dueDate", "message": "Due date must be in the future" }
    ]
  }
}

Query Parameters

Use query parameters for filtering, sorting, and pagination:

# Filtering
GET /api/tasks?status=pending
GET /api/tasks?status=pending&priority=high
GET /api/tasks?assignee=user123

# Sorting
GET /api/tasks?sort=dueDate        # Ascending
GET /api/tasks?sort=-dueDate       # Descending
GET /api/tasks?sort=-priority,dueDate  # Multiple fields

# Pagination
GET /api/tasks?page=2&limit=20
GET /api/tasks?offset=20&limit=20

# Field selection (sparse fieldsets)
GET /api/tasks?fields=id,title,status

# Search
GET /api/tasks?q=meeting
GET /api/tasks?search=urgent+deadline

# Date ranges
GET /api/tasks?createdAfter=2026-01-01&createdBefore=2026-01-31

Implementing REST in Express

const express = require('express');
const router = express.Router();

// GET /api/tasks - List tasks
router.get('/', async (req, res, next) => {
  try {
    const {
      status,
      assignee,
      sort = '-createdAt',
      page = 1,
      limit = 20
    } = req.query;

    // Build query
    const query = {};
    if (status) query.status = status;
    if (assignee) query.assigneeId = assignee;

    // Parse sort
    const sortOptions = {};
    sort.split(',').forEach(field => {
      if (field.startsWith('-')) {
        sortOptions[field.slice(1)] = -1;
      } else {
        sortOptions[field] = 1;
      }
    });

    // Execute with pagination
    const skip = (parseInt(page) - 1) * parseInt(limit);
    const [tasks, total] = await Promise.all([
      Task.find(query)
        .sort(sortOptions)
        .skip(skip)
        .limit(parseInt(limit)),
      Task.countDocuments(query)
    ]);

    res.json({
      data: tasks,
      meta: {
        total,
        page: parseInt(page),
        perPage: parseInt(limit),
        totalPages: Math.ceil(total / parseInt(limit))
      }
    });
  } catch (error) {
    next(error);
  }
});

// GET /api/tasks/:id - Get single task
router.get('/:id', async (req, res, next) => {
  try {
    const task = await Task.findById(req.params.id);
    if (!task) {
      return res.status(404).json({
        error: { code: 'NOT_FOUND', message: 'Task not found' }
      });
    }
    res.json({ data: task });
  } catch (error) {
    next(error);
  }
});

// POST /api/tasks - Create task
router.post('/', async (req, res, next) => {
  try {
    const task = new Task({
      ...req.body,
      createdBy: req.user.id
    });
    await task.save();

    res.status(201)
      .location(`/api/tasks/${task.id}`)
      .json({ data: task });
  } catch (error) {
    next(error);
  }
});

// PUT /api/tasks/:id - Replace task
router.put('/:id', async (req, res, next) => {
  try {
    const task = await Task.findByIdAndUpdate(
      req.params.id,
      { ...req.body, updatedAt: new Date() },
      { new: true, runValidators: true, overwrite: true }
    );

    if (!task) {
      return res.status(404).json({
        error: { code: 'NOT_FOUND', message: 'Task not found' }
      });
    }

    res.json({ data: task });
  } catch (error) {
    next(error);
  }
});

// PATCH /api/tasks/:id - Partial update
router.patch('/:id', async (req, res, next) => {
  try {
    const task = await Task.findByIdAndUpdate(
      req.params.id,
      { $set: { ...req.body, updatedAt: new Date() } },
      { new: true, runValidators: true }
    );

    if (!task) {
      return res.status(404).json({
        error: { code: 'NOT_FOUND', message: 'Task not found' }
      });
    }

    res.json({ data: task });
  } catch (error) {
    next(error);
  }
});

// DELETE /api/tasks/:id - Delete task
router.delete('/:id', async (req, res, next) => {
  try {
    const task = await Task.findByIdAndDelete(req.params.id);

    if (!task) {
      return res.status(404).json({
        error: { code: 'NOT_FOUND', message: 'Task not found' }
      });
    }

    res.status(204).end();
  } catch (error) {
    next(error);
  }
});

Middleware Patterns

What is Middleware?

Middleware functions sit between the incoming request and your route handlers. They can:

Execute code
Modify the request or response
End the request-response cycle
Call the next middleware in the stack

Request → [Middleware 1] → [Middleware 2] → [Route Handler] → Response
              │                 │                  │
              ▼                 ▼                  ▼
          Logging         Authentication      Business Logic

Common Middleware Patterns

Authentication Middleware

// middleware/authenticate.js
const jwt = require('jsonwebtoken');

async function authenticate(req, res, next) {
  try {
    // Extract token from header
    const authHeader = req.headers.authorization;
    if (!authHeader || !authHeader.startsWith('Bearer ')) {
      return res.status(401).json({
        error: { code: 'UNAUTHORIZED', message: 'No token provided' }
      });
    }

    const token = authHeader.substring(7);

    // Verify token
    const decoded = jwt.verify(token, process.env.JWT_SECRET);

    // Attach user to request
    const user = await User.findById(decoded.userId);
    if (!user) {
      return res.status(401).json({
        error: { code: 'UNAUTHORIZED', message: 'User not found' }
      });
    }

    req.user = user;
    next();
  } catch (error) {
    if (error instanceof jwt.TokenExpiredError) {
      return res.status(401).json({
        error: { code: 'TOKEN_EXPIRED', message: 'Token has expired' }
      });
    }
    res.status(401).json({
      error: { code: 'INVALID_TOKEN', message: 'Invalid token' }
    });
  }
}

// Optional authentication - doesn't fail if no token
function optionalAuth(req, res, next) {
  const authHeader = req.headers.authorization;
  if (!authHeader) {
    return next();  // Continue without user
  }

  // If token exists, validate it
  return authenticate(req, res, next);
}

Authorization Middleware

// middleware/authorize.js
function requireRole(...roles) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({
        error: { code: 'UNAUTHORIZED', message: 'Authentication required' }
      });
    }

    if (!roles.includes(req.user.role)) {
      return res.status(403).json({
        error: {
          code: 'FORBIDDEN',
          message: `Requires one of roles: ${roles.join(', ')}`
        }
      });
    }

    next();
  };
}

// Check ownership of resource
function requireOwnership(resourceFetcher) {
  return async (req, res, next) => {
    try {
      const resource = await resourceFetcher(req);
      if (!resource) {
        return res.status(404).json({
          error: { code: 'NOT_FOUND', message: 'Resource not found' }
        });
      }

      const ownerId = resource.userId || resource.createdBy || resource.owner;
      if (ownerId.toString() !== req.user.id.toString()) {
        return res.status(403).json({
          error: { code: 'FORBIDDEN', message: 'Not authorized to access this resource' }
        });
      }

      req.resource = resource;
      next();
    } catch (error) {
      next(error);
    }
  };
}

// Usage
router.delete('/:id',
  authenticate,
  requireOwnership(req => Task.findById(req.params.id)),
  taskController.delete
);

router.get('/admin/users',
  authenticate,
  requireRole('admin'),
  userController.list
);

Validation Middleware

// middleware/validate.js
const Joi = require('joi');

function validate(schema, property = 'body') {
  return (req, res, next) => {
    const { error, value } = schema.validate(req[property], {
      abortEarly: false,      // Return all errors
      stripUnknown: true,     // Remove unknown fields
      convert: true           // Type coercion
    });

    if (error) {
      const details = error.details.map(d => ({
        field: d.path.join('.'),
        message: d.message,
        type: d.type
      }));

      return res.status(400).json({
        error: {
          code: 'VALIDATION_ERROR',
          message: 'Request validation failed',
          details
        }
      });
    }

    // Replace with validated/sanitized value
    req[property] = value;
    next();
  };
}

// Schemas
const schemas = {
  createTask: Joi.object({
    title: Joi.string().min(1).max(200).required(),
    description: Joi.string().max(2000),
    dueDate: Joi.date().min('now'),
    priority: Joi.string().valid('low', 'medium', 'high').default('medium'),
    assigneeId: Joi.string().hex().length(24)
  }),

  updateTask: Joi.object({
    title: Joi.string().min(1).max(200),
    description: Joi.string().max(2000).allow(''),
    dueDate: Joi.date().allow(null),
    priority: Joi.string().valid('low', 'medium', 'high'),
    status: Joi.string().valid('pending', 'in_progress', 'completed')
  }).min(1),

  queryParams: Joi.object({
    page: Joi.number().integer().min(1).default(1),
    limit: Joi.number().integer().min(1).max(100).default(20),
    status: Joi.string().valid('pending', 'in_progress', 'completed'),
    sort: Joi.string()
  })
};

// Usage
router.post('/',
  authenticate,
  validate(schemas.createTask),
  taskController.create
);

router.get('/',
  validate(schemas.queryParams, 'query'),
  taskController.list
);

Error Handling Middleware

// middleware/errorHandler.js

// Custom error classes
class AppError extends Error {
  constructor(message, statusCode, code) {
    super(message);
    this.statusCode = statusCode;
    this.code = code;
    this.isOperational = true;
  }
}

class NotFoundError extends AppError {
  constructor(resource = 'Resource') {
    super(`${resource} not found`, 404, 'NOT_FOUND');
  }
}

class ValidationError extends AppError {
  constructor(message, details = []) {
    super(message, 400, 'VALIDATION_ERROR');
    this.details = details;
  }
}

class UnauthorizedError extends AppError {
  constructor(message = 'Authentication required') {
    super(message, 401, 'UNAUTHORIZED');
  }
}

class ForbiddenError extends AppError {
  constructor(message = 'Access denied') {
    super(message, 403, 'FORBIDDEN');
  }
}

// Error handling middleware
function errorHandler(err, req, res, next) {
  // Log error
  console.error('Error:', {
    message: err.message,
    stack: err.stack,
    url: req.url,
    method: req.method,
    userId: req.user?.id
  });

  // Handle known errors
  if (err.isOperational) {
    return res.status(err.statusCode).json({
      error: {
        code: err.code,
        message: err.message,
        details: err.details
      }
    });
  }

  // Handle Mongoose validation errors
  if (err.name === 'ValidationError') {
    const details = Object.values(err.errors).map(e => ({
      field: e.path,
      message: e.message
    }));
    return res.status(400).json({
      error: {
        code: 'VALIDATION_ERROR',
        message: 'Validation failed',
        details
      }
    });
  }

  // Handle Mongoose cast errors (invalid ObjectId)
  if (err.name === 'CastError') {
    return res.status(400).json({
      error: {
        code: 'INVALID_ID',
        message: `Invalid ${err.path}: ${err.value}`
      }
    });
  }

  // Handle duplicate key errors
  if (err.code === 11000) {
    const field = Object.keys(err.keyValue)[0];
    return res.status(409).json({
      error: {
        code: 'DUPLICATE_KEY',
        message: `${field} already exists`
      }
    });
  }

  // Unknown errors - don't leak details in production
  const message = process.env.NODE_ENV === 'production'
    ? 'An unexpected error occurred'
    : err.message;

  res.status(500).json({
    error: {
      code: 'INTERNAL_ERROR',
      message
    }
  });
}

// 404 handler for unknown routes
function notFoundHandler(req, res) {
  res.status(404).json({
    error: {
      code: 'ENDPOINT_NOT_FOUND',
      message: `Cannot ${req.method} ${req.url}`
    }
  });
}

Logging Middleware

// middleware/logger.js
const morgan = require('morgan');

// Custom token for user ID
morgan.token('user-id', (req) => req.user?.id || 'anonymous');

// Custom token for response time in a readable format
morgan.token('response-time-ms', (req, res) => {
  const time = morgan['response-time'](req, res);
  return time ? `${time}ms` : '-';
});

// Development logging
const developmentLogger = morgan('dev');

// Production logging (JSON format)
const productionLogger = morgan((tokens, req, res) => {
  return JSON.stringify({
    method: tokens.method(req, res),
    url: tokens.url(req, res),
    status: parseInt(tokens.status(req, res)),
    responseTime: parseFloat(tokens['response-time'](req, res)),
    contentLength: tokens.res(req, res, 'content-length'),
    userId: tokens['user-id'](req, res),
    userAgent: tokens['user-agent'](req, res),
    timestamp: new Date().toISOString()
  });
});

const logger = process.env.NODE_ENV === 'production'
  ? productionLogger
  : developmentLogger;

module.exports = logger;

Rate Limiting Middleware

// middleware/rateLimit.js
const rateLimit = require('express-rate-limit');
const RedisStore = require('rate-limit-redis');

// Basic rate limiter
const apiLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,  // 15 minutes
  max: 100,                   // 100 requests per window
  message: {
    error: {
      code: 'RATE_LIMIT_EXCEEDED',
      message: 'Too many requests, please try again later'
    }
  },
  standardHeaders: true,      // Return rate limit info in headers
  legacyHeaders: false
});

// Stricter limiter for authentication routes
const authLimiter = rateLimit({
  windowMs: 60 * 60 * 1000,   // 1 hour
  max: 5,                      // 5 attempts
  message: {
    error: {
      code: 'TOO_MANY_ATTEMPTS',
      message: 'Too many login attempts, please try again later'
    }
  }
});

// Usage
app.use('/api', apiLimiter);
app.use('/api/auth/login', authLimiter);

Middleware Order

Middleware order matters. A typical Express app might order middleware like this:

const express = require('express');
const app = express();

// 1. Security headers (first)
app.use(helmet());

// 2. CORS
app.use(cors(corsOptions));

// 3. Request parsing
app.use(express.json({ limit: '10mb' }));
app.use(express.urlencoded({ extended: true }));

// 4. Logging
app.use(logger);

// 5. Rate limiting
app.use('/api', apiLimiter);

// 6. Authentication (before protected routes)
app.use('/api', optionalAuth);

// 7. Routes
app.use('/api/auth', authRoutes);
app.use('/api/tasks', authenticate, taskRoutes);
app.use('/api/users', authenticate, userRoutes);

// 8. 404 handler (after routes)
app.use(notFoundHandler);

// 9. Error handler (last)
app.use(errorHandler);

Data Architecture

Data Modeling Fundamentals

Effective data modeling is crucial for application performance and maintainability.

MongoDB Document Design

MongoDB is a document database. Documents are BSON objects (like JSON):

// Task document
{
  _id: ObjectId("64a..."),
  title: "Complete API design",
  description: "Design RESTful endpoints for task management",
  status: "in_progress",
  priority: "high",
  dueDate: ISODate("2026-02-01"),
  createdAt: ISODate("2026-01-15"),
  updatedAt: ISODate("2026-01-20"),
  createdBy: ObjectId("64b..."),  // Reference to user
  assignee: {                      // Embedded document
    _id: ObjectId("64c..."),
    name: "John Doe",
    email: "john@example.com"
  },
  tags: ["api", "design"],         // Array
  comments: [                      // Array of embedded documents
    {
      _id: ObjectId("64d..."),
      text: "Looking good!",
      author: ObjectId("64e..."),
      createdAt: ISODate("2026-01-18")
    }
  ]
}

Embedding vs. Referencing

Approach	When to Use	Example
Embedding	Data accessed together, one-to-few relationships	Comments in a task
Referencing	Data accessed independently, one-to-many/many-to-many	Users and tasks

Embedding Example:

// Embedded comments - always fetched with task
const taskSchema = new mongoose.Schema({
  title: String,
  comments: [{
    text: String,
    authorId: mongoose.Schema.Types.ObjectId,
    authorName: String,  // Denormalized for display
    createdAt: { type: Date, default: Date.now }
  }]
});

// Adding a comment
await Task.findByIdAndUpdate(taskId, {
  $push: {
    comments: {
      text: "New comment",
      authorId: user._id,
      authorName: user.name
    }
  }
});

Referencing Example:

// Referenced assignee - fetched separately when needed
const taskSchema = new mongoose.Schema({
  title: String,
  assigneeId: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'User'
  }
});

// Populate when needed
const task = await Task.findById(taskId)
  .populate('assigneeId', 'name email');

Mongoose Schema Design

// models/Task.js
const mongoose = require('mongoose');

const taskSchema = new mongoose.Schema({
  title: {
    type: String,
    required: [true, 'Title is required'],
    trim: true,
    maxlength: [200, 'Title cannot exceed 200 characters']
  },
  description: {
    type: String,
    trim: true,
    maxlength: [2000, 'Description cannot exceed 2000 characters']
  },
  status: {
    type: String,
    enum: ['pending', 'in_progress', 'completed'],
    default: 'pending'
  },
  priority: {
    type: String,
    enum: ['low', 'medium', 'high'],
    default: 'medium'
  },
  dueDate: Date,
  completedAt: Date,
  createdBy: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'User',
    required: true
  },
  assigneeId: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'User'
  },
  projectId: {
    type: mongoose.Schema.Types.ObjectId,
    ref: 'Project'
  },
  tags: [String],
  metadata: {
    type: Map,
    of: String
  }
}, {
  timestamps: true,  // Adds createdAt and updatedAt
  toJSON: { virtuals: true },
  toObject: { virtuals: true }
});

// Indexes for common queries
taskSchema.index({ status: 1, dueDate: 1 });
taskSchema.index({ assigneeId: 1, status: 1 });
taskSchema.index({ projectId: 1, createdAt: -1 });
taskSchema.index({ createdBy: 1 });
taskSchema.index({ tags: 1 });

// Text index for search
taskSchema.index({ title: 'text', description: 'text' });

// Virtual property
taskSchema.virtual('isOverdue').get(function() {
  if (!this.dueDate || this.status === 'completed') return false;
  return new Date() > this.dueDate;
});

// Instance method
taskSchema.methods.complete = function() {
  if (this.status === 'completed') {
    throw new Error('Task is already completed');
  }
  this.status = 'completed';
  this.completedAt = new Date();
  return this.save();
};

// Static method
taskSchema.statics.findByAssignee = function(userId, options = {}) {
  const query = this.find({ assigneeId: userId });

  if (options.status) {
    query.where('status').equals(options.status);
  }

  return query.sort({ dueDate: 1 });
};

// Pre-save hook
taskSchema.pre('save', function(next) {
  if (this.isModified('status') && this.status === 'completed') {
    this.completedAt = new Date();
  }
  next();
});

const Task = mongoose.model('Task', taskSchema);
module.exports = Task;

Query Optimization

Using Indexes

Indexes dramatically improve query performance:

// Without index: Collection scan (slow)
db.tasks.find({ assigneeId: ObjectId("...") })

// With index: Index scan (fast)
taskSchema.index({ assigneeId: 1 });

Index Types:

// Single field index
taskSchema.index({ status: 1 });

// Compound index (order matters!)
taskSchema.index({ projectId: 1, createdAt: -1 });

// Text index for full-text search
taskSchema.index({ title: 'text', description: 'text' });

// Unique index
userSchema.index({ email: 1 }, { unique: true });

// Sparse index (only index documents with the field)
taskSchema.index({ assigneeId: 1 }, { sparse: true });

// TTL index (auto-delete documents)
sessionSchema.index({ createdAt: 1 }, { expireAfterSeconds: 3600 });

Query Patterns

// Efficient: Uses index
const tasks = await Task
  .find({ projectId, status: 'pending' })
  .sort({ dueDate: 1 })
  .limit(20)
  .select('title status dueDate');  // Only fetch needed fields

// Inefficient: Can't use index effectively
const tasks = await Task
  .find({ $or: [{ status: 'pending' }, { priority: 'high' }] })
  .sort({ randomField: 1 });

// Pagination with cursor (better for large datasets)
const tasks = await Task
  .find({ _id: { $gt: lastSeenId } })
  .sort({ _id: 1 })
  .limit(20);

// Aggregation pipeline for complex queries
const tasksByStatus = await Task.aggregate([
  { $match: { projectId: mongoose.Types.ObjectId(projectId) } },
  { $group: {
      _id: '$status',
      count: { $sum: 1 },
      avgPriority: { $avg: { $cond: [
        { $eq: ['$priority', 'high'] }, 3,
        { $cond: [{ $eq: ['$priority', 'medium'] }, 2, 1] }
      ]}}
    }
  },
  { $sort: { count: -1 } }
]);

Transactions

For operations that must succeed or fail together:

const session = await mongoose.startSession();
session.startTransaction();

try {
  // Create task
  const task = await Task.create([{
    title: 'New Task',
    projectId: project._id,
    createdBy: user._id
  }], { session });

  // Update project task count
  await Project.findByIdAndUpdate(
    project._id,
    { $inc: { taskCount: 1 } },
    { session }
  );

  await session.commitTransaction();
  return task[0];
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

Polyglot Persistence

What is Polyglot Persistence?

Polyglot persistence is the practice of using different data storage technologies for different data storage needs within a single application. Instead of forcing all data into one database type, you choose the best tool for each job.

When Different Databases Excel

Data Type	Best Fit	Examples
Structured, relational	PostgreSQL, MySQL	Users, orders, financial data
Document/flexible schema	MongoDB	Content, user profiles, product catalogs
Key-value/caching	Redis	Sessions, cache, real-time counters
Full-text search	Elasticsearch	Search, logging, analytics
Graph relationships	Neo4j	Social networks, recommendations
Time series	InfluxDB, TimescaleDB	Metrics, IoT, monitoring

Example: Multi-Database Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Application                               │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐         │
│   │   MongoDB    │  │    Redis     │  │ Elasticsearch│         │
│   │              │  │              │  │              │         │
│   │  - Tasks     │  │  - Sessions  │  │  - Search    │         │
│   │  - Projects  │  │  - Cache     │  │  - Logs      │         │
│   │  - Users     │  │  - Rate limit│  │  - Analytics │         │
│   │              │  │  - Pub/Sub   │  │              │         │
│   └──────────────┘  └──────────────┘  └──────────────┘         │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Implementing Polyglot Persistence

MongoDB for primary data:

// Primary data storage
const task = await Task.create({
  title: 'New Task',
  description: 'Task description',
  status: 'pending'
});

Redis for caching:

const Redis = require('ioredis');
const redis = new Redis(process.env.REDIS_URL);

// Cache user data
async function getUser(userId) {
  // Check cache first
  const cached = await redis.get(`user:${userId}`);
  if (cached) {
    return JSON.parse(cached);
  }

  // Fetch from database
  const user = await User.findById(userId);

  // Cache for 5 minutes
  await redis.setex(`user:${userId}`, 300, JSON.stringify(user));

  return user;
}

// Invalidate cache on update
async function updateUser(userId, data) {
  const user = await User.findByIdAndUpdate(userId, data, { new: true });
  await redis.del(`user:${userId}`);
  return user;
}

Redis for sessions:

const session = require('express-session');
const RedisStore = require('connect-redis').default;

app.use(session({
  store: new RedisStore({ client: redis }),
  secret: process.env.SESSION_SECRET,
  resave: false,
  saveUninitialized: false,
  cookie: {
    secure: process.env.NODE_ENV === 'production',
    httpOnly: true,
    maxAge: 24 * 60 * 60 * 1000  // 24 hours
  }
}));

Elasticsearch for search:

const { Client } = require('@elastic/elasticsearch');
const elastic = new Client({ node: process.env.ELASTICSEARCH_URL });

// Index task for search
async function indexTask(task) {
  await elastic.index({
    index: 'tasks',
    id: task._id.toString(),
    body: {
      title: task.title,
      description: task.description,
      status: task.status,
      tags: task.tags,
      createdAt: task.createdAt
    }
  });
}

// Search tasks
async function searchTasks(query, filters = {}) {
  const result = await elastic.search({
    index: 'tasks',
    body: {
      query: {
        bool: {
          must: [
            {
              multi_match: {
                query,
                fields: ['title^2', 'description', 'tags'],
                fuzziness: 'AUTO'
              }
            }
          ],
          filter: filters.status ? [
            { term: { status: filters.status } }
          ] : []
        }
      },
      highlight: {
        fields: {
          title: {},
          description: {}
        }
      }
    }
  });

  return result.hits.hits.map(hit => ({
    id: hit._id,
    ...hit._source,
    highlights: hit.highlight
  }));
}

// Keep Elasticsearch in sync with MongoDB
taskSchema.post('save', function(task) {
  indexTask(task).catch(console.error);
});

taskSchema.post('remove', function(task) {
  elastic.delete({
    index: 'tasks',
    id: task._id.toString()
  }).catch(console.error);
});

Challenges of Polyglot Persistence

Challenge	Mitigation
Data consistency	Event-driven sync, eventual consistency acceptance
Operational complexity	DevOps automation, monitoring
Query complexity	Clear service boundaries
Team knowledge	Training, documentation

Summary

This week covered backend and data patterns essential for building robust APIs:

RESTful API design uses resources, HTTP methods, and status codes for predictable interfaces
Middleware patterns handle cross-cutting concerns like authentication, validation, and error handling
MongoDB document design balances embedding and referencing based on access patterns
Indexes and query optimization are critical for performance
Transactions ensure data consistency for multi-document operations
Polyglot persistence uses the right database for each data type

These patterns form the foundation for Lab 4, where you’ll implement a complete backend with proper validation, error handling, and data access patterns.

Key Terms

REST: Representational State Transfer - architectural style for APIs
Resource: A noun that your API manages (users, tasks, projects)
Middleware: Functions that process requests in a pipeline
Idempotent: Operation that produces same result regardless of how many times executed
Document Database: Database storing data as flexible documents (MongoDB)
Index: Data structure that improves query performance
Polyglot Persistence: Using multiple database technologies in one application

Introduction

RESTful API Design

What is REST?

Resource-Oriented Design

HTTP Methods and CRUD Operations

HTTP Status Codes

Request and Response Design

Query Parameters

Implementing REST in Express

Middleware Patterns

What is Middleware?

Common Middleware Patterns

Authentication Middleware

Authorization Middleware

Validation Middleware

Error Handling Middleware

Logging Middleware

Rate Limiting Middleware

Middleware Order

Data Architecture

Data Modeling Fundamentals

MongoDB Document Design

Embedding vs. Referencing

Mongoose Schema Design

Query Optimization

Using Indexes

Query Patterns

Transactions

Polyglot Persistence

What is Polyglot Persistence?

When Different Databases Excel

Example: Multi-Database Architecture

Implementing Polyglot Persistence

Challenges of Polyglot Persistence

Summary

Key Terms

Further Reading