Redis is one of the most popular and versatile data-stores today, favored for its speed and simplicity. Redis offers features that resemble common data structures, such as strings, lists, hashes, sets, sorted sets, streams, and even geospatial indexes. Redis works by storing key-value pairs, such that a key names one value, and the value can be one of the aforementioned structures. Think of it like a directory, where you use a name (the key) to locate information about the person (the value).

For example, you could use a Redis hash to store user data by using a user_id key, then the value would be an object with hash fields mapped to values:

"user_123" : {
    "name": "Bob", 
    "age": 40, 
    "email": "bob@gmail.com"
  }

What makes Redis so Fast?

Leveraging Physical Attributes

While a variety of factors contribute to Redis’s speedy performance, the first one to understand is in-memory storage. All data in Redis lives in RAM; disk is used only for persisting data, never to serve a read. Think of RAM as your desk: limited space, but everything on it is within arm's reach. Meanwhile, disk storage is the filing room down the hall – far more capacity, but retrieving an item means getting up and walking.

To help quantify this advantage, DRAM access is ~100 ns, NVMe SSD is ~20-100 μs, and spinning disk (HDD) 5-10 ms. That makes DRAM ~200-1000x faster vs. NVMe and ~50,000-100,000x compared to HDD.

While tempting to conclude that Redis outperforms disk-backed databases like Postgres simply because it reads from RAM while the latter reads from disk, this doesn’t reflect how modern databases operate. Indeed, databases contain their own buffer pools (RAM managed directly by the DB) that store recently-accessed pages. Similarly, the kernel itself retains recently-read file blocks in RAM automatically, which the database can utilize. Taken together, even disk-backed databases do not primarily read from disk.

The real difference is the overhead required to support a disk representation, which is costly even when the disk is never read. A conventional database must parse and plan the query, consult a buffer manager to locate and pin the page, evaluate MVCC visibility, and deserialize the tuple from its on-disk format. Harizopoulos et al. (2008) benchmarked a conventional database with its data already fully cached in memory, and found that buffer management, latching, locking, and recovery consumed the large majority of instructions, leaving only a small fraction as useful query work. Redis has no on-disk format to translate from, so the structure it queries is the structure in memory.

Thus, our earlier analogy needs an amendment. A conventional database does keep frequently accessed pages on the desk; however, those pages are copies from the filing room, and the office enforces rules about them. Each time you want one, someone has to work out which file you actually need, look up where its copy is being held, and confirm that the version you have is the one you're supposed to read. By contrast, Redis's data is not a copy of anything. There is no filing room, no index, and no checkout procedure. What you see on the desk is the original, and only one.

While operating purely from memory contributes to Redis's speed, that alone does not explain what sets it apart; Redis also makes the most of low-level data structures.

Efficient Data Storage

Redis stores data incredibly efficiently, with small values being encoded in compact memory formats: listpack (which replaced ziplist in Redis 7.0), intset, and quicklist, that improve CPU cache locality. Hashes and sets are automatically converted to hash tables, lists become quicklists, and sorted sets become skiplists when exceeding the configured max size (Object Encoding docs).

To understand the advantage of these compact memory formats, we first must explore some computer architecture. Random access memory (RAM) is a massive array of bytes. Each byte has a number representing its position in the array, which also serves as the address. When the CPU runs a load instruction, it requests a small number of bytes at a specific address (eight, in the case of a pointer). However, the smallest unit that can be transferred from memory to the CPU is a 64-byte cache line. Memory is pre-divided into 64-byte blocks at fixed boundaries (0-63, 64-127, 128-191, …), so whichever block the address falls into is the one you get, along with everything else inside it.

Between CPU and RAM there are several levels of smaller and faster cache: L1 on the core, L2 beside it, and L3 shared across the chip. Each level is larger and slower than the last – ~1 ns for L1, a few nanoseconds for L2, ~15 ns for L3, and ~100 ns for main RAM. Indeed, a single trip to RAM amounts to about a hundred L1 hits, meaning that even once data is in memory where it sits can influence performance.

Now consider what happens when Redis reads a field from each structure. With a compact memory format such as a listpack, data is stored contiguously, meaning each field exists immediately after the previous. The first access costs a full trip to RAM, returning 64 bytes (most of the structure for a small collection). Since the scan moves forward through memory predictably, the CPU’s prefetcher proactively pulls in subsequent cache lines. Once the bytes are in L1, comparing each field costs ~1 ns, so twenty comparisons add ~20 ns.

On the other hand, a hash table stores its bucket array, entries, its key strings, and its value as separate allocations scattered across the heap. Each hop must be completed before the next address is available, making the four resulting trips to RAM dependent and non-overlapping. Thus, at small sizes, the O(N) scan outperforms the O(1) lookup.

However, the four trips involved in the hash table fetch are constant – they do not grow with the number of fields whereas the listpack’s scan does. After a certain size, the scan cost exceeds the cost of the RAM trips, and the structure stops fitting in the cache, making the comparisons themselves more expensive. Redis resolves this by automatically converting the compact structures into hash tables once they grow past the configured threshold.

In addition to clever utilization of data structures, architectural decisions further contribute to Redis's notable performance.

Single Threaded Architecture

Redis employs single-threaded command execution, avoiding the complexity and overhead associated with shared-state multithreaded systems: context switching, thread scheduling, lock contention, and even cache lines moving between cores. This design is viable because CPU is rarely the bottleneck for Redis, usually it is memory or network (FAQ). Since typical Redis commands are very small, adding the cost of coordination (which is roughly the same regardless of the operation size) becomes a bad tradeoff.

Of note, Redis is not strictly a single-threaded process – modern versions use threads for a variety of supporting functions, such as closing files, cleaning memory, or reading/writing to client sockets. Threaded I/O, introduced in Redis 6, parallelizes socket reads, writes and protocol parsing (redis.conf, THREADED I/O). However, only the main thread executes commands that touch the global keyspace, such as lookups, mutations, and triggering expiry/eviction to name a few. This architecture has the added benefit of preserving atomicity, since only a single thread is modifying Redis’s in-memory data at a given time.

What enables Redis to handle thousands of clients with a single thread is multiplexing and non-blocking I/O. Every client connection is a socket, and the kernel holds a receive buffer for each one.

Redis sets these sockets to non-blocking, so read and write operations return immediately if nothing is ready rather than blocking the thread. It then registers all of them with an event notification interface such as epoll, asking to be notified when one of them has something ready for it, then sleeps. A socket is ready when a read or write wouldn't halt execution: readable means that at least one byte has arrived, writeable means there's room in socket's send buffer. Because the kernel is the one putting arriving bytes into the buffers, it can add the socket to a ready list at that moment. When data arrives for one of those sockets, the kernel wakes Redis, which checks the short list of ready descriptors, runs the handler registered for each, and goes back to sleep. As a result, the thread never blocks on any individual client – it only pauses when no client has anything ready.

A helpful analogy is a restaurant with forty tables (client connections) and a single waiter (the Redis thread). If the waiter were to approach the table and wait until everyone was ready to order, this prevents the waiter from being able to attend other tables while they deliberate. Non-blocking here translates to the waiter asking first if a table is ready, and if not, he leaves. The issue is that the waiter now has to continuously walk the floor to determine which tables are ready, checking the whole room each lap, regardless of anyone actually is ready (this corresponds to select/poll). Meanwhile, epoll is akin to giving each table a call button that allows them to notify the waiter when they are ready. This allows the waiter to attend tables that are guaranteed to be ready without constantly checking all the tables.

It's worth noting an important drawback that this design introduces: a single long-running command will block all clients. The documentation highlights commands that involve many elements, such as SORT, LREM, SUNION. To troubleshoot this, the Diagnosing latency issues documentation provides a comprehensive guide.

Multiplexing and non-blocking I/O are part of the networking story, but two more factors facilitate Redis's network I/O efficiency.

Optimizations for Network I/O

Redis is further optimized for handling network I/O by using a custom protocol called Redis Serialization Protocol (RESP) and pipelining. RESP offers a simple implementation, fast parsing, and human readable commands, enabling Redis to quickly parse commands with minimal CPU cycles. This performance benefit is achieved by using prefixed lengths, thereby removing the need to scan for special characters or quote/escape the payload. Meanwhile, pipelining enables multiple commands to be sent with a single write operation by the client. The client can skip reading replies and continue to send commands to the query buffer. Redis then drains the query buffer and executes each command as it’s parsed, appending each reply to the client’s output buffer for the replies to go out together before Redis sleeps. This process reduces the latency by decreasing the total number of network round trips, and increases throughput by minimizing socket I/O, condensing multiple read() and write() syscalls into fewer (note that a read caps at 16KB, so a large pipeline may still require multiple syscalls).

Conclusion

There is no one thing that can take credit for Redis's performance. In reality it is the culmination of a deep understanding for how computers store and access memory combined with architectural decisions that maximize resource utilization based on Redis's use case. The fact that Redis operates entirely in-memory facilitates incredibly fast access and avoids costly operations involved in maintaining multiple data representations. Memory is further optimized by dynamically switching between compact memory formats and hash tables, which guarantees the best possible performance when working with smaller data structures. Moreover, as Redis is not bound by CPU and the operations are typically small, keeping all command execution on a single thread avoids unnecessary costs associated with shared-state, multi-threaded applications. Finally, networking optimizations such as a lightweight, custom communication protocol and pipelining minimize wasted CPU cycles without impacting the functionality.

Keep in mind that the refactoring solutions presented are not unique to the code smells presented and likely also appeared (or will appear) in other posts in this series. To me this illustrates the beauty of Fowler's Refactoring book, since he wrote it in a way that his solutions were reusable and the code smells simply reference them. However, for the sake of clarity and the ability for each article to be read as a standalone piece, I will likely reuse some prose and examples. I would also note that the problems the same refactor solves may be distinct, and understanding how the same pattern can apply to multiple code smells will strengthen your mental model and understanding of how to write "clean" code.

Links to other posts in this series:

Bloaters

Principles of Object-Oriented Programming

To frame this discussion of what violates the best practices of object-oriented programming (OOP) we first must understand the core principles that guide us.

Inheritance

Inheritance is the property that allows us to derive new classes by extending existing ones, thereby enabling code reuse. My favorite definition for inheritance comes from Deborah Armstrong's Quarks of Object Oriented Development (2006) as

a mechanism that allows the data and behavior of one class to be included in or used as the basis for another class

Consider an example where we have two classes representing animals:

class Dog:
    def __init__(self, name):
        self.name = name

    def walk(self):
        print("Walks on four legs")

    def bark(self):
        print("woof!")

class Cat:
    def __init__(self, name):
        self.name = name
    
    def walk(self):
        print("Walks on four legs")

    def meow(self):
        print("meow!")

We can simplify the above code by moving the shared properties into a parent class called Animal that Dog and Cat both inherit from:

class Animal:
    def __init__(self, name):
        self.name = name
    
    def walk(self):
        print("Walks on four legs")

class Dog(Animal):
    def bark(self):
        print("woof!")

class Cat(Animal):
    def meow(self):
        print("meow!")

This facilitates code reuse as we can easily create new animals, improves maintainability since changes to the Animal superclass only live in one location, and provides a built-in delegation mechanism such that subclasses can implement their own behaviors.

Encapsulation

The idea behind encapsulation is controlling what is accessible from outside an object. In other words, a clear division between an object's private internal state and its contract with (methods exposed to other objects/clients). In his 1986 paper Encapsulation and inheritance in object-oriented programming, Snyder posits that one should minimize the exposure of a module's implementation details to its clients to allow for changes made within the module without affecting the clients. This is particularly relevant to states within an object.

Imagine we introduce an energy state, indicating how energized the animal feels and some new methods that interact with it. Observe what happens when the state is left public (exposed in the interface):

class Animal:
    def __init__(self, name):
        self.name = name
        self.energy = 100     # public — anyone can set it to anything

    def walk(self):
        if self.energy < 10:
            print(f"{self.name} is too tired to walk")
            return
        self.energy -= 10
        print(f"{self.name} walks on four legs")

    def feed(self, amount):
        self.energy = min(100, self.energy + amount)

cat = Cat("Whiskers")
cat.energy = 5000    # nonsense, but nothing prevents it
cat.walk()           # energy is now 4990
cat.energy = -20     # the cat now has negative energy??

To prevent this from occurring we encapsulate energy (the Python convention is to prefix with _ ) so that it can only be accessed within the class. Then we create and expose special method for the public interface:

class Animal:
    def __init__(self, name):
        self.name = name
        self._energy = 100

    # ... walk and feed implementation

    @property
    def energy(self):
        return self._energy

Note that the combination of the @property decorator with no setter is what truly blocks assignment– the leading underscore is just convention.

We can even change the derivation of energy within the class without affecting the way the client interacts with the object at all:

class Animal:
    def __init__(self, name):
        self.name = name
        self._activity = 0

    @property
    def energy(self):
        return max(0, 100 - self._activity) # now derived

Polymorphism

Subtype polymorphism is what enables a method to use a uniform interface regardless of the specific type. This is best illustrated with an example.

Consider a variation of the example we've been working with using a generic speak method that is concretely implemented by the subclasses:

class Animal:
    def __init__(self, name):
        self.name = name
    
    def walk(self):
        print("Walks on four legs")
    
    def speak(self):
        raise NotImplementedError 

class Dog(Animal):
    def speak(self):
        print("woof!")

class Cat(Animal):
    def speak(self):
        print("meow!")

The speak method can now be called on any Animal. The client does not know or care about specific details as those are handled within the subclasses. This relates back to the Liskov substitution principle, which posits that an object of the superclass (Animal) can be replaced by an object of the subclass (Dog or Cat) without breaking the program.

Abstraction

Abstraction is the mechanism that ties everything together and allows us to provide simplified models (classes) for complex realities to suppress irrelevant details. In this regard, a class can be thought of as an abstraction of an object, such that the class encapsulates data and behavior, and inheritance is what allows the encapsulated data and behavior to be based on an existing class (Armstrong, 2006). Finally, polymorphism enables different objects to respond to the same invocations and respond appropriately.

Repeated Switches

This is another way of saying if/else blocks that appear in more than one location in the system. The primary concern is that changes or additions made to any one switch statement must occur in every instance of the repeated switch.

What makes this smell an Object-Orientation Abuser is that the repeated switch approach neglects the elegant solution of polymorphism offered by object-oriented design, in which we significantly improve the readability and organization of the code.

Refactoring Solutions

Replace Conditional with Polymorphism

This technique is actually quite similar to the Strategy Pattern (which I previously wrote about). The goal is to eliminate large conditional branches by replacing them with classes and polymorphism. Consider the logic we use to assign a dough, which involves adding a different flour for each distinct type:

def make_dough(dough_variant): 
    if variant == "regular": 
        add_flour() 
    if variant == "gluten-free": 
        add_almond_flour() 
    # New variant: whole wheat 
    if variant == "whole wheat": 
        add_whole_wheat_flour()

Instead of repeating the conditional or decomposing it, we instead convert it to it's own type, letting each concrete implementation handle the implementation details:

class Dough:
    def add_flour(self):
        # Implemented by specific types

class RegularDough(Dough):
    def add_flour(self):
        add_flour()

class GlutenFreeDough(Dough):
    def add_flour(self):
        add_almond_flour()

class WholeWheatDough(Dough):
    def add_flour(self):
        add_whole_wheat_flour()

def make_dough(dough):
    activate_yeast()
    add_olive_oil()
    dough.add_flour()
    # ... rest of steps

Now instead of adding a conditional branch whenever we need to add a new variant, we can simply add a child to the Dough class, then define the add_flour method.

Temporary Field

The temporary field describes a field that is only set sometimes, adding ambiguity to the object or function containing it. The solution involves a sequence of three strategies that build upon each other.

Consider the following example of a PizzaOrder class:

class PizzaOrder:
    def __init__(self, items, is_delivery):
        self.items = items
        self.is_delivery = is_delivery
        self.delivery_address = None
        self.delivery_fee = None     
        self.driver_tip = None        

    def total(self):
        subtotal = sum(item.price for item in self.items)
        if self.is_delivery:
            return subtotal + self.delivery_fee + self.driver_tip
        return subtotal

The primary issue here is that three of the variables (delivery_address, delivery_fee, and driver_tip) are dead weight when the order is not a delivery.

Refactoring Solution

Extract Class

First, these inconsistent variables need a dedicated place to live, such as a dedicated Delivery class:

class Delivery:
    def __init__(self, address, fee, tip):
        self.address = address
        self.fee = fee
        self.tip = tip

Move Function

Next, the behavior related to those fields (computing the subtotal) is moved to a method within the new class:

class Delivery:    
    # ... init    
    def cost(self):
        return self.fee + self.tip

This allows us to refactor the original PizzaOrder total method to the following:

class PizzaOrder(self):
    def total(self):
        subtotal = sum(item.price for item in self.items)
        if self.delivery:
            return subtotal + self.delivery.cost()
        return subtotal

The combination of these two changes enforce the single-responsibility principle since the knowledge of the delivery cost lives with the delivery data instead of bleeding into the order, and improve encapsulation by exposing a single cost method to handle the implementation details.

Introduce Special Case

Fowler proposes one additional step, which is to create a new class to handle the conditional. Introduce Special Case involves creating a class to capture common behaviors to replace special-case checks, and is commonly referred to as the Null Object pattern because null is frequently the value involved in special-case processing (Fowler, p.289). In the PizzaOrder scenario, we would introduce a special class NoDelivery to handle the case when delivery is null:

class NoDelivery:                  # the special case
    def cost(self):
        return 0

Then, instead of our guard checking if self.delivery is truthy, we can replace the variable assignment with self.delivery = delivery or NoDelivery(), which will guarantee that self.delivery.cost() is always callable.

Alternative Classes with Different Interfaces

This smell involves two or more classes that essentially do the same job but have different method names or signatures, thereby preventing client code from using them interchangeably despite being conceptually the same. Take the following example where two similar classes are used to notify customers that their pizza is on the way using a notify_customer method:

class EmailNotifier:
    def send_email(self, address, subject, body):
        print(f"Emailing {address}: {subject}")

class SmsNotifier:
    def dispatch_text(self, phone_number, message):
        print(f"Texting {phone_number}: {message}")

def notify_customer(notifier, customer):
    if isinstance(notifier, EmailNotifier):
        notifier.send_email(customer.email, "Pizza Update", "On its way!")
    elif isinstance(notifier, SmsNotifier):
        notifier.dispatch_text(customer.phone, "Pizza on its way!")

Refactoring Solutions

Change Function Declaration

This refactoring technique encompasses both renaming the method itself and modifying the parameters (Fowler, p.125).

The first step is to align the method names (we'll use notify). Next, look at both the declarations and match up the signatures:

class EmailNotifier:
    def notify(self, recipient, message):
        print(f"Emailing {recipient}: {message}")

class SmsNotifier:
    def notify(self, recipient, message):
        print(f"Texting {recipient}: {message}")

Now that both classes contain the same general interface, the client no longer cares which type it receives– it can invoke them interchangeably!

def notify_customer(notifier, customer):
    notifier.notify(customer.contact, "Pizza on its way!")

Move Function

Move function comes into play when the alternative classes have somewhat different capabilities (but the core functionality remains the same). For example, let's say the EmailNotifier implementation had a record_sent method to log each notification to the company's audit log:

class EmailNotifier:
    def notify(self, recipient, message):
        print(f"Emailing {recipient}: {message}")
        self.record_sent(recipient, message)

    def record_sent(self, recipient, message):
        audit_log.append((recipient, message))

class SmsNotifier:
    def notify(self, recipient, message):
        print(f"Texting {recipient}: {message}")
        # no logging here — the caller has to do it

def notify_customer(notifier, customer):
    notifier.notify(customer.contact, "Pizza on its way!")
    if isinstance(notifier, SmsNotifier):
        audit_log.append((customer.contact, "Pizza on its way!"))

We can move the logging behavior into the SmsNotifier to ensure consistency between the classes, keeping the client code lean:

class EmailNotifier:
    def notify(self, recipient, message):
        print(f"Emailing {recipient}: {message}")
        self.record_sent(recipient, message)

    def record_sent(self, recipient, message):
        audit_log.append((recipient, message))

class SmsNotifier:
    def notify(self, recipient, message):
        print(f"Texting {recipient}: {message}")
        self.record_sent(recipient, message)

    def record_sent(self, recipient, message):
        audit_log.append((recipient, message))

def notify_customer(notifier, customer):
    notifier.notify(customer.contact, "Pizza on its way!")

Extract Superclass

While not necessary, taking this a step further to make the code more extensible in the future might involve pulling the shared notify interface into one Notifier superclass so that additional notifiers can be added in the future.

Refused Bequest

Refused Bequest describes a two ways in which a subclass can violate the inheritance contract (Fowler, p.84):

1. Subclass only uses some of the methods and/or fields inherited from its parent.

class Enemy:
    def __init__(self, name, hp):
        self.name = name
        self.hp = hp

    def take_damage(self, amount):
        self.hp -= amount
    
    def move(self):
        print(f"{self.name} moves toward you")

    def flee(self):
        print(f"{self.name} runs away")

class Boss(Enemy):
    def take_damage(self, amount):
        self.hp -= amount
    
    def move(self):
        print(f"{self.name} moves toward you")
    # a Boss never flees, so it just ignores the inherited flee()

1. Subclass that reuses behavior but fails to support the interface of the superclass. This occurs when the subclass inherits a method that it cannot actually fulfill (meaning it throws an error or returns something nonsensical).

class Enemy:
    # ... init and take_damage from before
    def move(self):
        print(f"{self.name} moves toward you")

    def flee(self):
        print(f"{self.name} runs away")

class Turret(Enemy):
    def move(self):
        raise NotImplementedError("turrets can't move")

    def flee(self):
        raise NotImplementedError("turrets can't flee") 

While Fowler considers the former case a trivial issue often not worth cleaning up, he emphasizes the severity of the latter violation. This smell also represents a violation of the Liskov Substitution Principle and the promise of inheritance as we can no longer guarantee that any subclass of Enemy can be substitued without breaking the program.

Refactoring Solutions

Push Down Method / Push Down Field

To resolve the tension of the mild case, the solution involves moving the unused methods and fields out of the parent and down into the siblings that actually use them. Say, taking flee out of Enemy and wiring it into the subclasses that actually use it, preventing the Boss subclass from ever receiving it.

Replace Subclass with Delegate

Rather than break the inheritance contract, the fix is to hold a reference to the superclass, then extract the parts it wants.

class Turret:
    def __init__(self, name, hp):
        self._enemy = Enemy(name, hp)
    
    def take_damage(self, amount):
        self._enemy.take_damage(amount)

    def attack(self):
        print(f"{self._enemy.name} fires from a fixed position")

    # omitted move() and flee()

The tradeoff here is that we are forwarding the boilerplate code (take_damage and any other methods we delegate) from the Enemy class in order to achieve a valid contract.

Replace Superclass with Delegate

This mis-inheritance also crops up on the parent's side when we drag an entire interface into a subclass only to use a portion of it. Fowler provides the example of making a stack a subclass of list, causing all of the list operations to be forwarded to stack despite most not applying. The solution is to make a new stack class with the list a property that we can delegate operations to (Fowler, p.399). Continuing with the video game example, imagine we have a Sprite class that we use for rendering drawings. We might logically attempt to make Enemy a subclass of Sprite in order to inherit the draw method:

class Sprite:
    def __init__(self, image):
        self.image = image
        self.x = 0
        self.y = 0

    def draw(self):
        print(f"drawing {self.image} at ({self.x}, {self.y})")

    def set_position(self, x, y):
        self.x, self.y = x, y


class Enemy(Sprite):   # inherits Sprite to reuse drawing
    def __init__(self, name, hp, image):
        super().__init__(image)
        self.name = name
        self.hp = hp

The issue is that an Enemy is not truly a kind of Sprite, it simply uses one to render on the screen. Instead, we let Enemy hold a reference to a Sprite, delegating the behavior we need (with the caveat of forwarding boilerplate from before):

class Enemy:
    def __init__(self, name, hp, image):
        self.name = name
        self.hp = hp
        self._sprite = Sprite(image)      

    def draw(self):
        self._sprite.draw()   # delegate the rendering 

    def move_to(self, x, y):
        self._sprite.set_position(x, y)

Final Thoughts

This category of code smells emphasizes how the core principles of OOP (inheritance, encapsulation, polymorphism, abstraction) operate in tandem to reduce waste and enforce a robust, maintainable codebase. One thing that I took away is the utility of Extract Class, not only in how it addresses encapsulation and the single-responsibility principle, but how it facilitates a shift in how I view code. I can now recognize when I'm "cross-contaminating" or overloading a class or function, and can more proactively refactor before it becomes a significant lift.

Additionally, the Refused Bequest smell got me thinking about contracts between objects in a way I hadn't considered before. While you can write code that compiles and runs despite violating an inheritance contract, it plants seeds for a future errors and forces the developer (or future reader) to perform more work trying to unravel why something broke. Further, the contract that is being broken is one that the compiler cannot see. While the compiler can enforce a method signature (e.g., invalid types, wrong number of arguments), the behavioral contracts (what a method promises to do) are invisible to it. Indeed, this is dangerous because it defers the cost and separates the failure point from the location of the code.

References:

1. Fowler, M. (2018). Refactoring: Improving the design of existing code (2nd ed.). Addison-Wesley Professional

2. Mäntylä, M.V., Lassenius, C. Subjective evaluation of software evolvability using code smells: An empirical study. Empir Software Eng 11, 395–431 (2006). https://doi.org/10.1007/s10664-006-9002-8

3. Black, A. Object-oriented programming: Some history, and challenges for the next fifty years. Information and Computation 231, 3-20 (2013). https://doi.org/10.1016/j.ic.2013.08.002

4. Snyder, A. Encapsulation and Inheritance in Object-Oriented Programming Languages. (1986). https://dl.acm.org/doi/10.1145/960112.28702

5. Armstrong, D. The Quarks of Object-Oriented Development. (2006). https://doi.org/10.1145/1113034.1113040

Links to other posts in this series:

Object-Orientation Abusers

Long Function

This code smell arises when a method or function is simply too large. The basis for this smell is that the longer a function is, the more difficult it is to reason about. Since modern languages have mostly eliminated overhead costs associated with in-process calls, there is significant value in breaking a large function up into smaller functions. Furthermore, the key to easily understandable small functions is deliberate and descriptive naming, which serves to reduce the burden placed on the reader caused by switching context to see what the function does. A good rule of thumb is to write a function whenever you feel the need to comment something, such that the function contains the code that you wanted to comment and is named after the intention (Fowler, 2018, p.73).

Refactoring Solutions

Extract Function

This is the first strategy to reach for when trying to shorten a function. The goal is to take a fragment of code that shares a similar purpose, and then extract it into its own aptly named function.

For example, let's say you wanted to make a pizza, which involves 3 main steps:

1. Make the dough (regular or gluten-free)

2. Assemble the pizza

3. Bake the pizza

We could combine it all into one function as follows (apologies if I missed some steps if there are any pizza experts here 🥲):

def make_pizza(dough_variant):
    # Make the dough
    activate_yeast()
    add_olive_oil()
    if dough_variant == "regular":
        add_flour()
    if dough_variant == "gluten-free":
        add_almond_flour()   
    knead_dough()
    rise_dough()
    toss_dough()
    
    assemble_pizza()
    
    bake_pizza()

We could extract all the steps for making the dough into its own function that we call within the make_pizza(variant) method:

def make_dough(variant):
    activate_yeast()
    add_olive_oil()
    if variant == "regular":
        add_flour()
    if variant == "gluten-free":
        add_almond_flour()        
    knead_dough()
    rise_dough()
    toss_dough()

def make_pizza(dough_variant):
    make_dough(dough_variant)
    
    assemble_pizza()
    
    bake_pizza()

Encapsulating the irrelevant implementation details makes it much easier to reason about what the make_pizza(variant) function is doing. We also now have the added benefit of reusability, such that the make_dough(variant) function can be called elsewhere if we were to extend the functionality (e.g., different types of pizza that use the same dough). The caveat with this technique is that the names must be clear and descriptive to minimize the effort spent discerning the purpose of a code fragment (Fowler, 2018, p.107).

In some cases there may be many parameters and temporary variables that make it difficult to extract a function as the result would require forwarding all that context to the new method. Fowler provides a number of options for dealing with these based on the specific obstacle.

Replace Temp with Query

Let's expand the first example to have make_pizza use a temporary variable bake_temp that it passes to the bake_pizza method based on the desired "crispiness" (another new parameter) of the pizza:

def make_pizza(dough_variant, crispiness):
    make_dough(dough_variant)
    if crispiness == "well-done":
        bake_temp = 500
    if crispiness == "regular":
        bake_temp = 450

    assemble_pizza()
    
    bake_pizza(bake_temp)

This is a problem because now anytime we want to extract something that requires that context, we have to duplicate the logic. Instead, we turn it into a query (a simple method) that contains the logic and returns the value we need.

def bake_temp_for(crispiness):
    if crispiness == "well-done":
        return 500
    if crispiness == "regular":
        return 450
    

def make_pizza(dough_variant, crispiness):
    make_dough(dough_variant)
    assemble_pizza()
    
    bake_pizza(bake_temp_for(crispiness))

Introduce Parameter Object

Fowler introduces the idea of a data clump as a groups of items that consistently appear together (Fowler, 2018, p.140). In the pizza example, as the make_pizza method grows more realistic (in the sense that it accounts for all the factors that actually matter for making a pizza), we end up having to pass more and more data to our extracted helper functions. We actually can see this happening as we extend the functionality to account for things like crispiness. What would happen if we were to introduce a few more factors, such as size, sauce, and toppings? We would definitely need to consider the size when adding sauce and toppings because it determines the amount used. Similarly, the size determines the quantity of ingredients that go into making our dough, and how long the pizza bakes for.

def make_pizza(dough_variant, crispiness, size, sauce, toppings):
    make_dough(dough_variant, size)
    assemble_pizza(size, sauce, toppings)
    bake_pizza(bake_temp_for(crispiness), size)

You can already see how this is becoming tedious. Instead, what if we were to bundle all the features into a single order object:

class PizzaOrder:
    def __init__(self, dough_variant, crispiness, size, sauce, toppings):
        self._dough_variant = dough_variant
        self._crispiness = crispiness
        # ... You get the point

def make_pizza(order):
    make_dough(order)
    assemble_pizza(order)
    bake_pizza(order)

We now have a much more concise make_pizza function that only takes in a single argument, from which we can extract the properties we need in each helper function. Fowler argues for passing the whole record (what he calls "Preserve the Whole Object") rather than deriving select values for a few reasons (Fowler, 2018, p.318):

1. Protects against future changes. For example, what happens when we need to extract more values, such a cheese variable from the PizzaOrder

2. It reduces the size of the parameter list, which makes the function more comprehensible.

3. If there are many functions called with the parts, that logic can be moved to the object so it can be invoked by the helper functions.

Replace Function with Command

If none of the above strategies successfully resolve the interdependent local variables and parameters, Fowler recommends switching to what he calls the heavy artillery: Replace Function with Command (Fowler, 2018, p.73).

In this strategy, you convert the entire function into an object, where the temporary variables become fields and the sub-steps become methods that share state. Consider the following PizzaMaker class to replace the make_pizza function:

class PizzaMaker:
    def __init__(self, order):
        self.order = order

    def make(self):
        self._make_dough()
        self._assemble()
        self._bake()

    def _make_dough(self):
        # Steps to make the dough
    def _assemble(self):
        # Steps to assemble the pizza
    def _bake(self):
        # Steps to bake the pizza

While this is not a perfect example because it's particularly long or complicated, hopefully you can see the goal of breaking down the function into methods and fields contained within a class. We can leverage the properties provided by classes such as shared state to create methods that manipulate parameters, eliminating the need to pass around variables from the main function to helpers.

The strategies above aim to resolve excessively large functions by extracting code fragments into logical groupings and clearing obstacles that prevent us from doing that cleanly. The next few strategies provide ways to remove conditionals and loops, which are often good candidates for extraction.

Decompose Conditional

The essence of this technique is to delegate the specifics of conditional logic into its own clearly named function that makes the calling code easier to read and reason about. Imagine our hypothetical pizza shop is running a promotion where a large pizza is half off from 2-3 pm (a pizza happy hour, if you will).

def calculate_pizza_cost(order):
    if (2pm ≤ datetime.now() ≤ 3pm): # Pseudocode
        return order.pizza.cost * 0.5
    else:
        return order.pizza.cost
    
        

Instead, we can delegate this logic to a new clearly named function that gives us tells us which rate we should use. We can even decompose the legs so that the evaluation reads more clearly:

def is_pizza_happy_hour():
    if (2pm ≤ datetime.now() ≤ 3pm): # Pseudocode
        return True
    else:
        return False

def happy_hour_cost(order):
    return order.pizza.cost * 0.5

def regular_cost(order)
    return order.pizza.cost

def calculate_pizza_cost(order):
    if is_pizza_happy_hour():
        return happy_hour_cost(order)
    else:
        return regular_cost(order
    

This allows the reader to quickly grasp the purpose of the calculate_pizza_cost function without getting caught up trying to trace the branching logic of the conditional.

Replace Conditional with Polymorphism

This strategy is actually quite similar to the Strategy Pattern (which I previously wrote about). The goal is to eliminate large conditional branches by replacing them with classes and polymorphism. Consider the logic we use to assign a dough, which involves adding a different flour for each distinct type:

def make_dough(dough_variant):
    if variant == "regular":
        add_flour()
    if variant == "gluten-free":
        add_almond_flour()
    # New variant: whole wheat
    if variant == "whole wheat":
        add_whole_wheat_flour()

Instead of repeating the conditional or decomposing it, we instead convert it to it's own type, letting each concrete implementation handle the implementation details:

class Dough:
    def add_flour(self):
        # Implemented by specific types

class RegularDough(Dough):
    def add_flour(self):
        add_flour()

class GlutenFreeDough(Dough):
    def add_flour(self):
        add_almond_flour()

class WholeWheatDough(Dough):
    def add_flour(self):
        add_whole_wheat_flour()

def make_dough(dough):
    activate_yeast()
    add_olive_oil()
    dough.add_flour()
    # ... rest of steps

Now instead of adding a conditional branch whenever we need to add a new variant, we can simply add a child to the Dough class, then define the add_flour method.

Split Loop

Fowler posits that loops should be extracted into their own methods, and further separated into distinct functions (Fowler, 2018, p.227). He gives the example of a loop that updates two variables on each iteration (stepping away from the pizza example because it doesn't make much sense here):

people = [person_a, person_b, person_c, ...]
average_age = 0
total_salary = 0
for person in people:
    average_age += person.age
    total_salary += person.salary

average_age = average_age / len(people)

# Applying the Split Loop refactor, this would become:
total_salary = 0
for person in people:
    total_salary += person.salary

average_age = 0
for person in people:
    average_age += person.age

average_age = average_age / len(people)

While it does slightly increase the amount of code written and force us to execute the loop twice, it removes the ambiguity of what the loop is actually doing, which allows us to clearly describe each one and place it into it's own method that can replace the code in the larger function. Fowler emphasizes the importance of separating refactoring from optimization, noting that we can easily recombine the loops if it turns out to be a bottleneck in the future (Fowler, 2018, p.228).

def total_salary():
    total_salary = 0
    for person in people:
        total_salary += person.salary
    return total_salary

def average_age():
    average_age = 0
    for person in people:
        average_age += person.age
    return average_age

Long Parameter List

First, let's understand the purpose of a parameter list:

"The parameter list to a function should summarize the points of variability of the function, indicating the primary ways in which that function may behave differently" (Fowler, 2018, p.324).

This smell appears when a function requires a large amount of data and requires an extensive parameter list to be passed in. In such cases, it becomes challenging to reason about the key ways a function might differ.

Refactoring Solutions

Replace Parameter with Query

Reach for this strategy when an argument passed into the function can be easily derived by the function itself. The idea is to shift the responsibility of determining the value from the calling code to the function itself. Fowler notes that this technique should not be used if removing the parameter adds an unwanted dependency to the function (i.e., an element that the function should be ignorant of). The ideal case is when the value you want to remove from the function arguments is retrieved by querying another parameter in the list.

Consider the following Order class

class Order:
    def __init__(self, quantity, item_price):
        self._quantity = quantity
        self._item_price = item_price
        self._discount_level = 2 if quantity > 100 else 1
    
    def final_price(self):
        base_price = self._quantity * self._item_price
        return discounted_price(base_price, self._discount_level)

    def discounted_price(base_price, discount_level):
        if discount_level == 1:
            return base_price * 0.95
        elif discount_level == 2:
            return base_price * 0.9
        

Rather than pass in self._discount_level to the method, we can remove it from the parameter list and simply derive it in the function:

    def discounted_price(base_price):
        if self._discount_level == 1:
            return base_price * 0.95
        elif self._discount_level == 2:
            return base_price * 0.9

Preserve Whole Object

In the event where you are passing several individual properties of an object as arguments, consider passing the whole entity instead, then extracting only what you need. Take the example below, where we pass in each parameter from our order individually:

def make_pizza(toppings, sauce, dough_variant, crispiness, cheese):
    make_dough(dough_variant)
    # ... you get the point by now


order = Order(
    toppings = ['pepperoni', 'bell peppers'],
    sauce = 'regular', 
    dough_variant = 'regular',
    crispiness = 'well-done',
    cheese =  'fresh mozzarella'
)

make_pizza(order.toppings, order.sauce, order.dough_variant, order.crispiness, order.cheese)

Alternatively, we simply pass in the whole object, then extract what we need:

def make_pizza(order):
    make_dough(order.dough_variant)
    # ... you get the point by now


order = Order(
    toppings = ['pepperoni', 'bell peppers'],
    sauce = 'regular', 
    dough_variant = 'regular',
    crispiness = 'well-done',
    cheese =  'fresh mozzarella'
)

make_pizza(order)

Introduce Parameter Object

If this looks familiar... that's because it is. It is in fact the same exact strategy recommended for reducing long functions, and is what would enable us to perform the refactor shown above.

Remove Flag Argument

A flag argument refers to an argument the caller passes to indicate which behavior to execute. Fowler avoids flag arguments as they make it harder to understand what function calls are available and how to call them (Fowler, 2018, p.315). To illustrate the problem, consider the the following line of code: make_pizza(order, True)

What is True doing here? There's no way to know what it's doing without opening the make_pizza and reading through the body– it silently switches the behavior of the function but tells the reader nothing.

Consider the following example where we use an is_rush flag:

def make_pizza(order, is_rush):
    make_dough(order)
    assemble_pizza(order)
    if is_rush:
        express_bake(order)
    else:
        standard_bake(order)

In this case, the flag is more descriptive, but the reader still has to open the make_pizza method and read through it to understand what behavior change is actually taking place. Instead, we remove the flag argument, replacing it with explicit, intention-revealing functions:

def make_rush_pizza(order):
    make_dough(order)
    assemble_pizza(order)
    express_bake(order)

def make_regular_pizza(order):
    make_dough(order)
    assemble_pizza(order)
    standard_bake(order)

And if you wanted to eliminate the duplicate body:

def make_rush_pizza(order):
    _make_pizza(order, express_bake)

def make_regular_pizza(order):
    _make_pizza(order, standard_bake)

def _make_pizza(order, bake_step):
    make_dough(order)
    assemble_pizza(order)
    bake_step(order)

Note that bake_step is not a flag being decoded, but a function that is passed from the public entry points (make_rush_pizza and make_regular_pizza)

While there is a time and place for flag arguments, such as avoiding writing explicit functions for every combination of flag values, they serve as an indicator that a function is doing too much.

Combine Functions into Class

This technique works well when multiple functions share several parameter values. The idea is to create a class containing the common fields. Consider the following example where multiple steps involved in making the pizza share some fields:

def make_dough(size, dietary_restrictions, dough_variant):
    # Logic for making the dough

def assemble_pizza(size, dietary_restrictions, toppings, sauce, cheese):
    # Logic for assembling the pizza

def bake_pizza(size, dietary_restrictions, crispiness):
    # Logic for baking the pizza

Noticing that all three of these functions share two of the same parameters, we create a PizzaMaker class that encapsulates the common data (i.e., make the shared arguments fields):

class PizzaMaker:
    def __init__(self, size, dietary_restrictions):
        self.size = size
        self.dietary_restrictions = dietary_restrictions

    def make_dough(self, dough_variant):
    # uses self.size, self.dietary_restrictions, and dough_variant

    def assemble_pizza(self, toppings, sauce, cheese):
    # uses self.size, self.dietary_restrictions, and the rest

    def bake_pizza(self, crispiness):
    # uses self.size, self.dietary_restrictions, and crispiness

Now observe how this simplifies the client code:

# Before
make_dough(size, dietary_restrictions, dough_variant)
assemble_pizza(size, dietary_restrictions, toppings, sauce, cheese)
bake_pizza(size, dietary_restrictions, crispiness)

# After
maker = PizzaMaker(size, dietary_restrictions)
maker.make_dough(dough_variant)
maker.assemble_pizza(toppings, sauce, cheese)
maker.bake_pizza(crispiness)

This solution falls into a Long Parameter List fix since the shared parameters were the bloat, and moving them into fields shrinks all the signatures simultaneously. It also sets us up for later refactoring, since any helper that is defined inside the class can freely use self.size and self.dietary_restrictions. If you were paying attention, you might point out that we could further refactor this by combining all shared fields into a single Order object that is initialized in the PizzaMaker class, from which the helper functions can derive what they need.

Additionally, while this resembles the solution to Replace Function with Command from the previous section, note that they are solutions answer to solve different problems. In the original example, we created a class to break down an excessively large function into an object so that the internal methods could share state. In the present example, we used a class to encapsulate shared data and reduce the size of the parameter lists and simplify the calling code.

Large Class

Classes that attempt to do too much, and often contain too many fields

Refactoring Solutions

Extract Class

This technique involves bundling related variables into their own components. To demonstrate, imagine our Order object from before contained information about both the pizza and the individual who placed the order.

class Order:
    def __init__(toppings, dough_variant, sauce, ..., customer_name, customer_phone_number, customer_address):
        self.toppings = toppings
        self.customer_name = customer_name        
        # ...
        

At this point we can take the customer information and bundle it into a Customer class, then pass the entire object into the Order class:

class Customer:
    def __init__(name, phone_number, address):
        self.name = name
        self.phone_number = phone_number
        self.address = address

class Order:
    def __init__(toppings, dough_variant, sauce, ..., customer)
        self.toppings = toppings
        self.customer = customer

Extract Superclass

This refactoring strategy leverages the power of inheritance to consolidate similarities among multiple classes into a superclass. Imagine you are creating creatures for a video game, such as "werewolf" and "undead"

class Werewolf: 
    def __init__(self, name, hp, mp, strength):
        self.name = name
        self.hp = hp
        self.mp = mp
        self.strength = strength

    def beast_claw(self):
        return f"{self.name} attacked for {self.strength} damage" 

class Undead:
    def __init__(self, name, hp, mp, intelligence):
        self.name = name
        self.hp = hp
        self.mp = mp
        self.strength = strength

    def rot_breath(self):
        return f"{self.name} attacked for {self.intelligence} damage"

    

We can combine these into a common superclass which extracts the fields and methods that are shared and leaves what differs to the subclasses. To do this, we apply what Fowler (2018) describes as Pull Up Field (p.353) and Pull Up Method (p.350). In this example, the methods used to attack have different names and bodies, but represent the same intent. With an additional helper function to get attack_power and a small refactor we can also move a generic attack function to the superclass using what Fowler labels the Form Template Method (Fowler, 2018, p.351):

class Enemy:
    def __init__(self, name, hp, mp):
        self.name = name
        self.hp = hp
        self.mp = mp

    def attack(self):
        return f"{self.name} attacked for {self.attack_power()} damage"

    def attack_power(self):
        raise NotImplementedError   # each creature supplies its own

class Werewolf(Enemy):
    def __init__(self, name, hp, mp, strength):
        super().__init__(name, hp, mp)
        self.strength = self.strength

    def attack_power(self):
        return self.strength

class Undead(Enemy):
    def __init__(self, name, hp, mp, intelligence):
        super().__init__(name, hp, mp)
        self.intelligence = intelligence

    def attack_power(self):
        return self.intelligence
    

Data Clumps

Data clumps describe entities that are frequently found together, such as a group of fields in multiple classes or parameters in several methods.

Refactoring Solutions

Data clumps can be dealt with by employing three refactors we have already seen before and go hand-in-hand. Once you've identified the common fields, group them into an object (Extract Class) to provide a layer of abstraction and separate concerns. This makes it easier to maintain since the data that changes together or are dependent on each other are now found together. If the data clump occurs in a method, you can pass the newly created class as a parameter (Introduce Parameter Object), reducing the bloated signature and allowing you to extract what you need in the function itself (Preserve Whole Object).

Primitive Obsession

Primitives are the most basic, built-in data types offered by a language, such as strings, arrays, integers, and booleans. The essence of this smell is an overuse of primitive types as opposed to developing your own simple classes to enforce special behaviors.

Refactoring Solutions

Replace Primitive with Object

Fowler (2018) comments that any time he wants to do more than simply printing an object, he creates a new class (p.174). This provides the developer with a clear place to add behavior specific to the data. Consider an integer crispiness level parameter in a pizza baking function.

def bake_pizza(order, crispiness):
    if crispiness < 1 or crispiness > 5:        # validation here
        raise ValueError("crispiness must be 1-5")
    minutes = 8 + crispiness * 2                # conversion logic here
    ...

def print_receipt(order, crispiness):
    labels = {1: "Soft", 3: "Golden", 5: "Extra Crispy"}   # formatting here
    print(f"Crispiness: {labels.get(crispiness)}")

While we can introduce guards and raise exceptions to enforce the behavior we want (integer between 1 and 5, bake time proportional to crispiness), the integer cannot validate itself, name itself, or tell you its bake time. To clean up this function, we can gather up all that information into a Crispiness object that implements all of that behavior:

class Crispiness:
    def __init__(self, level: dict):
        if not 1 <= level <= 5:
            raise ValueError("crispiness must be 1-5")
        self.level = level

    def bake_minutes(self):
        return 8 + self.level * 2

    def label(self):
        return {
            1: "Soft", 
            3: "Golden", 
            5: "Extra Crispy"
        }.get(self.level, "Medium") # Levels 2 and 4 default to "Medium"

# Revised function passing in whole object
def bake_pizza(order, crispiness):
    bake_for(crispiness.bake_minutes())

def print_receipt(order, crispiness):
    print(f"Crispiness: {crispiness.label()}")

Replace Type Code with Subclasses

A type code is a primitive field that determines what kind of thing an object is. Extending the game creatures from before, we might initialize the Enemy class with a creature_type type code:

class Enemy:
    def __init__(self, name, hp, mp, creature_type):
        self.name = name
        self.hp = hp
        self.mp = mp
        self.creature_type = creature_type # "werewolf" or "undead" — the type code

    def attack(self):
        if self.creature_type == "werewolf":
            return f"{self.name} slashes with claws"
        elif self.creature_type == "undead":
            return f"{self.name} breathes rot"

    def weakness(self):
        if self.creature_type == "werewolf":
            return "silver"
        elif self.creature_type == "undead":
            return "fire"

With this approach, we end up having to recreate the if-else branching logic for every method. This means an addition of a new enemy would require updating every single conditional (Fowler calls this smell "Repeated Switches"). The result is much simpler methods that leave the concrete implementation details to the subclasses to prevent bloated methods and duplicate code:

class Enemy:
    def __init__(self, name, hp, mp):
        self.name = name
        self.hp = hp
        self.mp = mp

class Werewolf(Enemy):
    def attack(self):
        return f"{self.name} slashes with claws"
    def weakness(self):
        return "silver"

class Undead(Enemy):
    def attack(self):
        return f"{self.name} breathes rot"
    def weakness(self):
        return "fire"

Takeaways

Something I noticed is that the book itself is written following Fowler's own refactoring principles. The names are clear and the refactoring strategies are designed to solve singular problems. He even breaks down smaller operations into their own strategies, such as "Encapsulate Variable" (p. 132) and "Rename Field" (p.244). The sheer amount of information contained in Fowler's book is overwhelming, but I do appreciate his consistency. While it's difficult at first when trying to digest the countless smells and refactors, the way he builds up each strategy help quickly create a mental model and solidify patterns.

References:

1. Fowler, M. (2018). Refactoring: Improving the design of existing code (2nd ed.). Addison-Wesley Professional

2. Mäntylä, M.V., Lassenius, C. Subjective evaluation of software evolvability using code smells: An empirical study. Empir Software Eng 11, 395–431 (2006). https://doi.org/10.1007/s10664-006-9002-8

Writing clean, maintainable code is crucial in developing software that stands the test of time. In today's landscape where there is a massive push for AI-generated code, I suspect the art of producing readable code will become even more valuable as entire codebases are built and refactored using large language models. Furthermore, as this code and other AI-generated content pollutes the internet, it will becomes increasingly difficult for new engineers to identify "good" code. As such, I'm committed to understanding what is constitutes truly robust code. Moreover, the introduction of AI-assisted coding has minimized the need for the junior engineer, since senior engineers can now delegate more trivial tasks and code generation to an LLM. The danger, which likely won't be realized for several years, is that there is no longer a built-in path for new engineers to receive guidance and mentorship to give them the skills required to serve more senior roles. I suspect the consequences will become apparent over the next 10-20 years when senior and staff engineers retire and there are no suitable replacements. However, that won't stop the oversaturation of the current job market for entry-level software engineers. While companies may realize this and take efforts to alleviate the problem by pushing for more junior headcount, I would rather do everything in my power to write code like a senior engineer than wait around for companies to take action.

Hence, the introduction of my "Developing as a Developer" series. In each part, I plan to explore and apply a concept that will teach me (and anyone who reads it) to produce better code. While I'm going to do my best to keep each part narrowly scoped, the nature of software development is that several topics are inherently linked. For example, originally I wanted to only discuss code smells in this article, but quickly realized that the accompanying refactoring techniques are crucial in building a mental model of what constitutes clean code. Then I realized that I should probably investigate SOLID principles before getting into code smells, and now my original two part series has quickly become who-knows-how-many parts.

With that said, one constant in life and computer science is change. Before delving into the specific topics, I think it's important to understand the history of the ideas and how they have changed or evolved over time. Tracing the origins of the ideas serves two main purposes:

1. It exercises our critical thinking muscles. We can speculate on reasons for change and improve our intuition of what will work/not work. We get to learn how subject matter experts adopt and adapt theories. This is important because blindly following principles will not make you a better programmer. In Martin's words: "principles have to be applied with judgement. If they are applied by rote it is just as bad as if they are not applied at all" (Martin, 2009). Furthermore, historizicing ideas gives us context into how theories are applied by different practitioners. Through investigating the origins of modern principles we develop a mutual understanding of their foundational underpinnings and the shared language to communicate and critique them. Without this, we would end up realizing Martin's concerns.

2. Proper attribution of content and ideas. A huge issue I've noticed (which has been exacerbated by the rise of LLM-generated content) is failing to properly cite sources. People may copy and paste or paraphrase a response from generative AI without ever knowing (or caring) where that information came from. That's fucked up. With time these types of posts flood the internet and it will grow increasingly difficult to trace the origin of knowledge.

History of Code Smells

Refactoring describes "change[s] made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior" (Fowler & Beck, 2018, p.45). While they knew that refactoring was valuable for writing maintainable and scalable software, both authors recognized there was a gap in communicating about refactoring beyond the vague notion of programming aesthetics. Thus, when writing the first edition of their book in the 90's, Martin Fowler and Kent Beck sought to clarify how to identify when code needs to be refactored. The conception of "smells" (attributed to Kent Beck, perhaps inspired by his newborn daughter) refers to an indication (not definitive proof) that some refactoring may be needed (Fowler & Beck, 2018, p. 71). Fowler and Beck outline 24 distinct code smells in the 2nd edition of their book, which represent common patterns and structures that likely benefit from refactoring. Not all instances of code that needs refactoring will fit cleanly into one of the named categories, but familiarity with the smells will point you in the right direction.

In their 2003 paper titled "A taxonomy and initial empirical study of bad smells in code," Finnish researchers at Helsinki University of Technology proposed grouping the original 22 code smells into 7 different classifications to make them more understandable and clarify relationships between the smells (Mäntylä et al., 2003):

Bloaters: Something in the code that has grown out of control and can no longer be maintained.

Object-Orientation Abusers: Cases where the solution fails to leverage object-oriented design effectively.

Change Preventers: Code structures that when changed result in a cascade of other changes on dependent structures.

Dispensables: Redundant or otherwise unnecessary code.

Encapsulators: Issues pertaining to the chain of delegation in how objects hide or pass along data.

Couplers: Code structures are too tightly coupled.

Other Smells: Smells that don't fit into any of the other categories.

Later in 2006, Mäntylä refined the taxonomy down to five groups in, collapsing two of the categories related to coupling (Encapsulators was absorbed by Couplers).

Since the taxonomy was published before the 2nd edition of Refactoring, there does not exist a perfect mapping to pull from the literature. Of note, a few code smells were renamed from the 1st to the 2nd edition:

• Long Method ➡️ Long Function

• Switch Statements ➡️ Repeated Switches

• Lazy Class ➡️ Lazy Element

• Inappropriate Intimacy ➡️ Insider Trading (hmm, I wonder why).

Four new smells related to newer concerns (such as code comprehension, shared state, immutability, and opting for first-class functions over loops) were also added in the 2nd edition: Mysterious Name, Global Data, Mutable Data, and Loops. For the sake of completeness and consistency, I will consider these part of a grouping I'll call "New Smells".

The following table provides a brief description and the category for each of the 24 smells that I will be investigating in more depth throughout this series.

Link to Code Smell – Bloaters article

History of SOLID Design Principles

SOLID represents "five principles intended to improve the make source code more understandable, flexible, and maintainable" (SOLID, Wikipedia). These principles were first articulated across the late-1990s Engineering Notebook columns for the C++ Report and later consolidated in an abbreviated form in Robert C. Martin's 2000 paper Design Principles and Design Patterns, in which he discusses the concept of software rot: the process of a once-beautiful architecture degrading into an unmanageable mess. Interestingly, it was not Martin himself who formed the mnemonic device– that is attributed to Michael Feathers, who noted the opportunity to rearrange the letters via an email to Martin around 2004 (Martin, 2018, p.58).

Of note, two principles predate Martin:

Open-Closed Principle

The Open-Closed Principle originates from Bertrand Meyer's (1988) book Object-Oriented Software Construction, where he states that "modules should be both open and closed" (Meyer, 1997) (I can't seem to find a copy of the 1st edition, but he has published the 1997 2nd edition for free on his website, so I will reference that instead). A module is considered open if it's still available for extension (e.g., adding fields to the data structures or expanding its functionality). A module is closed if it is available for use by other modules and has been given a well-defined, stable description (Meyer, 1997). Meyer (1997) resolved the open-closed tension through inheritance: a concrete base class stays closed to modification while new functionality is added by subclassing it (pp.58-60).

Martin's version of the Open-Closed Principle maintains the slogan of "open for extension but closed for modification" but differs in the mechanism. Martin leverages abstraction– client code depends on a fixed abstract type, and the system is extended by deriving new implementations of that type, protecting existing code from future edits. This shift was likely due to issues related to implementation inheritance, whereby a subclass inherits the actual implementation (including code and state), resulting in tight coupling between child and the internal workings of the parent, thus violating encapsulation (Snyder, 1986; Gamma et al., 1994). Furthermore, Mikhajlov and Sekerinski (1998) highlight the fragility of the paradigm by demonstrating how revising a base class can result in breaking subclasses. Inheriting from an abstraction solves these issues by decoupling the implementation. The subtype only inherits the method signatures (name, arguments, return type) and behavioral promise (conditions for calling the method, guarantees once it returns, rules about state) and is responsible for supplying its own behavior.

Liskov Substitution Principle

The Liskov Substitution Principle was introduced by MIT's Barbara Liskov in her 1987 OOPSLA Keynote, titled Data Abstraction and Hierarchy, where she discusses the importance of data abstraction in developing programs that are highly maintainable and easy to modify when requirements change (Liskov, 1987, p.33). When discussing a type hierarchy composed of subtypes and supertypes, she proposes the following substitution property:

If for each object o1, of type S there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1, is substituted for o2, then S is a subtype of T.

(Liskov, 1987, p.25).

In other words, an object of a superclass can be replaced by an object of a subclass without breaking the program.

The concept was formally introduced into academia by Liskov and Wing's 1994 paper A Behavioral Notion of Subtyping, in which they argued that "the objects of the subtype ought to behave the same as those of the supertype as far as anyone or any program using supertype objects can tell" (Liskov & Wing, 1988).

SOLID Principles: Current State

Martin describes the principles more definitively in his later books Agile Software Development: Principles, Patterns and Practices, and Agile Principles, Patterns, and Practices in C#. Since then, SOLID principles have become a standard practice in the field for creating scalable and maintainable software, with several academic publications supporting their usage (Ali, 2002; Cabral et al., 2024; Yanakiev et al., 2025).

Below lists the SOLID principles from Martin (2002) that will be explored at length in a future article:

Single Responsibility Principle (SRP) (p.95)

A class should have only one reason to change"

Open-Closed Principle (OCP) (p.99)

"Software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification"

Liskov Substitution Principle (LSP) (p.111)

"Subtypes must be substitutable for their base types"

Interface Segregation Principle (ISP) (p.138)

"Clients should not be forced to depend on methods that they do not use"

Dependency Inversion Principle (DIP) (p.127)

"a. High-level modules should not depend on low-level modules. Both should depend on abstractions.

b. Abstractions should not depend on details. Details should depend on abstractions."

References:

1. Fowler, M. (2018). Refactoring: Improving the design of existing code (2nd ed.). Addison-Wesley Professional

2. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.

3. Mäntylä, Mika & Vanhanen, Jari. (2003). Bad Smells in Software – a Taxonomy and an Empirical Study.

4. Mäntylä, M.V., Lassenius, C. Subjective evaluation of software evolvability using code smells: An empirical study. Empir Software Eng 11, 395–431 (2006). https://doi.org/10.1007/s10664-006-9002-8

5. Martin, Robert C, (2000). Design Principles and Design Patterns

6. Raphael Cabral, Marcos Kalinowski, Maria Teresa Baldassarre, Hugo Villamizar, Tatiana Escovedo, and Hélio Lopes. 2024. Investigating the Impact of SOLID Design Principles on Machine Learning Code Understanding. https://doi.org/10.1145/3644815.3644957

7. Mohamed Ali, Azrajabeen. (2022). The Impact of SOLID Principles on Code Quality and Software Lifecycle. 7. 10.5281/zenodo.15062293

8. Yanakiev, I., Lazar, B. M., & Capiluppi, A. (2025). Applying SOLID principles for the refactoring of legacy code: An experience report. Journal of Systems and Software, 220, 112254

9. Martin, Robert (2018). Clean Architecture: A Craftsman's Guide to Software Structure and Design. Pearson. p. 58. ISBN 978-0-13-449416-6

10. Meyer, B. (1997). Object-Oriented Software Construction (2nd ed.) Prentice Hall.

11. Liskov, B. (1987). "Data Abstraction and Hierarchy." OOPSLA '87 (pub. SIGPLAN Notices 23(5), 1988).

12. Liskov, B. H., & Wing, J. M. (1994). A behavioral notion of subtyping. ACM Transactions on Programming Languages and Systems (TOPLAS), 16(6), 1811-1841.

13. Cook, W. R., Hill, W. L., & Canning, P. S. (1990). Inheritance is not subtyping. In Proceedings of the 17th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (pp. 125–135)

14. Mikhajlov, Leonid; Sekerinski, Emil (1998). A study of the fragile base class problem

15. Snyder, A. (1986). "Encapsulation and Inheritance in Object-Oriented Programming Languages." OOPSLA '86, pp. 38–45.

To start, let's look at the definition provided in Design Pattern Elements of Reusable Object-Oriented Software:

“Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from the clients that use it”

(Gamma, Helm, Johnson, & Vlissides, p. 315).

To better understand the definition, let's break down the components of the Strategy Design Pattern:

• Client: Determines what strategy to use and passes it to the context; interacts with the context exclusively.

• Context: Applies the strategy selected by the client; holds a reference to the strategy object.

• Strategy Interface: The common interface all concrete strategies must implement; declares a method the context invokes to execute a strategy

• Concrete Strategies: Encapsulates specific logic of each behavior/algorithm; can be swapped out based on requirements.

Now let’s break down each portion of that definition:

“family of algorithms”

This refers to a related group of algorithms that can be selected or switched dynamically depending on the conditions (Behavioral Design Patterns). Consider each set of behaviors as a “family” of algorithms. For example, let’s say you wanted to switch between different database storage methods. You could choose from MongoDB, Postgres, or MySQL, all of which belong to the "storage" family. Similarly, if you wanted to switch between different API endpoints in a cryptocurrency tracker application. You could choose from CoinGecko, CoinMarketCap, or a "mock" endpoint that simulates an HTTP response for testing. These algorithms would all belong to the "data retrieval" family.

“encapsulates each one”

For a given family of algorithms, we can invoke different behaviors by interacting only with the interface for that family, leaving the implementation details to classes we define for each behavior. Continuing with the data retrieval example above, we might have an interface with a get_latest_price(symbol) method that delegates functionality to a specific data retrieval algorithm to get the latest price for a given cryptocurrency symbol. Given our interface DataStrategy, our Concrete Strategies are the specific variations: CoinGeckoStrategy, CoinMarketCapStrategy, and MockStrategy, each of which implement their own get_latest_price() method.

“lets the algorithm vary independently from clients that use it”

To "vary independently" is another way of saying the client and the algorithm are decoupled, meaning either side (client or strategy) can change without forcing a change on the other side. You could update the implementation of one retrieval algorithm without ever touching the calling code and vice-versa. For example, say you needed to update a JSON field in your CoinGecko strategy, or update the base URL. You can update the function that normalizes the data into your internal response shape, or edit HTTP call without ever opening the client code. On the other side, say your client code needs to be modified to balance load and fail over across two different APIs to account for rate limits, whereas previously it simply set one strategy. You could achieve this by wrapping existing the strategies in a new one, enabling you to implement a new feature without touching the context or any of the existing strategies.

Below I've provided a diagram to help visualize the strategy set up described above:

What problems does the Strategy Pattern Solve?

From a readability perspective, the Strategy pattern eliminates conditional statements by encapsulating behavior to separate Strategy objects (Gamma, Helm, Johnson, & Vlissides, pp. 317-318), which serves to prevent unruly, monolithic code blocks. For example, if we were to implement the data retrieval strategies naively, it would look something like this:

class PriceFetcher:
    def get_latest_price(self, symbol: str, fetch_strategy: str):

        if fetch_strategy == "CoinGecko":
            api_url = "" # CoinGecko's URL
            headers = {} # Required headers
            # ...
        elif fetch_strategy == "CoinMarketCap":
            api_url = "" # CoinMarketCap URL
            headers = {} # Required headers           
            # ...
        
        elif fetch_strategy == "Mock":
            return (
                {
                    "name": "Bitcoin",
                    "symbol": "btc",
                    "price": 10000,
                    "timestamp": datetime.now(timezone.utc),
                    "currency": "USD",
                },
            ) 
    
        else:
            print("Error: Unsupported fetch strategy")

It's clear how this could become a problematic class when more conditions are added or specific implementation details are modified. Furthermore, this implementation violates two principles of software engineering:

Single Responsibility Principle

Single Responsibility Principle. This principle states that a class should only have one reason to change (Martin, 2002, p. 95). In a later article, Martin specifies that a reasons for change are people, as they are the drivers of changes (Martin, 2014). In the example above, there are three different parties that might require changes for different reasons: The CoinGecko team, the CoinMarketCap team, and you (the developer). The first two are out of our control, such as changes to the base URL, required headers, or endpoints. However, changes driven by any of the parties requires an edit to the same method in the naive PriceFetcher . If the CoinGecko team renames a JSON field, you're editing the same method that also contains the CoinMarkerCap's auth logic, which has nothing to do with the change.

We can remedy this by assigning each condition its own strategy, ensuring the individual strategies have exactly one reason to change (its own vendor). The responsibility of the PriceFetcher class is to orchestrate the execution of the strategies, not manage the specific implementation details of each strategy. Thus, only changes related to the orchestration, such as adding caching, modifying error handling, or changing which source is preferred should elicit changes.

(Polymorphic) Open Closed Principle

Open/Closed Principle. This principle states that classes should be open for extension, but closed for modification (Martin, 2002, p. 99). In other words, when the requirements change, you should be able to extend the functionality by adding new code, rather than altering old code that already works. In the naive example above, adding a new new fetch strategy (say, to a third endpoint) would require opening the PriceFetcher class and adding a new elif block.

Putting it all together

The Strategy pattern implementation mirrors Martin's description of creating "abstractions that are fixed and yet represent an unbounded group of possible behaviors" (Martin, 2002, p.100). Refer to the following table to see how the Strategy pattern relates to Martin's Open/Closed Principle:

To achieve this, we start by creating the base class and derivatives:

# --- Base Class --- #
class FetchStrategy(ABC):
    @abstractmethod
    def fetch_all_prices(self) -> list[dict]:
        raise NotImplementedError("Subclass must implement this method")


# --- Concrete Strategies --- #
class CoinGeckoStrategy(FetchStrategy):
    def fetch_all_prices(self) -> list[dict]:
        api_url = "https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd"
        headers = {"x-cg-demo-api-key": os.getenv("COINGECKO_API_KEY")}
        response = requests.get(api_url, headers=headers)
        data = response.json()
        formatted_prices = []
        for coin in data:
            curr_coin = {
                "name": coin["name"],
                "symbol": coin["symbol"],
                "price": coin["current_price"],
                "timestamp": datetime.now(timezone.utc),
                "currency": "USD",
            }
            formatted_prices.append(curr_coin)

        return formatted_prices


class CoinMarketCapStrategy(FetchStrategy):
    def fetch_all_prices(self) -> list[dict]:
        api_url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest"
        headers = {
            "Accept": "application/json",
            "X-CMC_PRO_API_KEY": os.getenv("COINMARKETCAP_API_KEY"),
        }
        response = requests.get(api_url, headers=headers)
        json = response.json()
        data = json["data"]
        formatted_prices = []
        for coin in data:
            curr_coin = {
                "name": coin["name"],
                "symbol": coin["symbol"].lower(),
                "price": coin["quote"]["USD"]["price"],
                "timestamp": datetime.now(timezone.utc),
                "currency": "USD",
            }
            formatted_prices.append(curr_coin)
        return formatted_prices

class MockStrategy(FetchStrategy):
    def fetch_all_prices(self) -> list[dict]:
        return [
            {
                "name": "Bitcoin",
                "symbol": "btc",
                "price": 10000,
                "timestamp": datetime.now(timezone.utc),
                "currency": "USD",
            },
            {
                "name": "CannoliCoin",
                "symbol": "ctc",
                "price": 999999,
                "timestamp": datetime.now(timezone.utc),
                "currency": "USD",
            },
        ]

Next, define the PriceFetcher context.

class PriceFetcher:
    def __init__(self):
        self._fetch_strategy: FetchStrategy | None = None

    def set_fetch_strategy(self, strategy: FetchStrategy):
        self._fetch_strategy = strategy

    def fetch_all_prices(self) -> list[dict]:
        return self._fetch_strategy.fetch_all_prices()

Now for the client code:

price_fetcher = PriceFetcher()
# Set a strategy before calling fetch_all_prices()
price_fetcher.set_fetch_strategy(CoinGeckoStrategy()) 
latest_crypto_prices = price_fetcher.fetch_all_prices()
print(latest_crypto_prices)

Taken together, the behavior of the system is open for extension (we could add a new derivative to extend the functionality) but closed for modification (no changes needed to PriceFetcher or the FetchStrategy base class when a derivative is added).

Drawbacks

No software engineering topic is complete without a discussion of trade-offs, because at the end of the day there are no perfect solutions. The following are summarized from the Strategy Pattern section of Design patterns: Elements of reusable object-oriented software (Gamma, Helm, Johnson, & Vlissides), which I recommend checking out for further reading since I only scratch the surface here:

1. A client must be aware of the nuances of the strategies in order to select the appropriate one. For example, how would the calling code know whether to use CoinGecko or CoinMarketCap? While this is a non-issue in the demo since both strategies return the same standardized list of dictionaries, making the choice arbitrary. However, if we needed to account for operational logistics, such as rate limits, pricing tiers, coin coverage, or reliability, that knowledge needs to live in the client as selecting the appropriate strategy is the client's responsibility.

2. Since the Strategy base class is shared by all ConcreteStrategy classes regardless of complexity, there will likely be information passed through the interface that is never utilized, meaning additional overhead is created for no benefit. An example of this is in the MockStrategy, which would accept symbol as a parameter in the get_latest_price method but always returns the same response.

3. More objects, classes, and interfaces in the application. The naive single-class fetcher built with chained if/elif statements is only a single object. Meanwhile, three objects (one for each concrete strategy), a base class, and a context class were required to implement the Strategy pattern in the demo provided.

Final Thoughts

The Strategy Pattern is a powerful tool for encapsulating interchangeable behaviors and letting them vary independently of the code that uses them. There are clear organizational benefits: giving each algorithm its own class adheres to the Single Responsibility Principle (each strategy only has one reason to change), which keeps the code maintainable and simpler to reason about as the codebase grows. It also embodies Martin's polymorphic reading of the Open/Closed Principle, where further changes are achieved by adding new code, not by modifying working code (Martin, 2002, p. 100), making system more resilient. However, these benefits come at a cost: more objects, communication overhead via a shared interface, and clients need to understand how strategies differ to choose the appropriate one. The Strategy pattern may not be the best option when variations are few and stable, or the behavioral logic in each condition is trivial. simple. In such cases a single-class approach would avoid creating unnecessary objects and abstractions. Indeed, the true skill is judging which dimensions of the system are likely to change, and only implementing the Strategy Pattern when there are meaningfully different variants for the dimension.

References:

1. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Addison-Wesley.

2. Martin, Robert C. (2014). "The Single Responsibility Principle". The Clean Code Blog.

3. Martin, R. C. (2002). Agile Software Development, Principles, Patterns, and Practices. Prentice Hall. ISBN 978-0135974445.

By default, Python code is executed synchronously, meaning that once the function is called, the calling code blocks execution of the main thread until the function finishes. If the function raises an exception, the calling code can wrap the call site with a try/except block to handle the errors. While this is convenient and makes it easier to reason about the code we write, there may be times when we want to utilize more of our available resources, or handle more tasks at once. Enter, concurrency and parallelism:

Concurrency: The ability of a system to handle multiple tasks by allowing overlapping (but not necessarily simultaneous) execution.

Parallelism: The simultaneous execution of multiple tasks that utilize multiple processing units. (Divakaran). This requires a multicore CPU, multiple CPUs, a GPU, or multiple computers in a cluster (Ramalho, 2022, p.697)

While the two concepts are similar in that they involve the execution of multiple tasks, note that parallelism requires simultaneous execution. Concurrency is the umbrella term for structuring a program to complete more than one task at once, and parallelism is a specific form, or subset of concurrency. All parallel systems are concurrent, but not all concurrent systems are parallel (Ramalho, p. 695).

I/O-bound tasks vs. CPU-bound tasks

When discussing different types of concurrent programming, it's important to understand the distinction between I/O-bound (input/output) and CPU-bound tasks. A program is said to be CPU-bound if the bottleneck is the CPU's processing power, which occurs in problems involving significant CPU computation (often times operations requiring complex math processing). Examples include: audio or image processing, computer vision, and machine or deep learning. Meanwhile, a task is considered I/O-bound when the bottleneck is the communication between the program and the outside world. Common examples include: reading data from a user input or file, making a network request, interacting with external devices, or database operations.

Processes vs. Threads

Another important distinction is that of a process and a thread. For these I found the most helpful definitions came from Fluent Python (Ramalho, 2022):

Process: An instance of a computer program in execution, containing it's own memory space and a portion of the CPU time. Processes are isolated entities that contain their own memory spaces. Processes communicate via pipes, sockets, or memory mapped files by converting Python objects into of raw bytes to pass from one process to another. A drawback with processes is the overhead involved in communication, and fact that not all Python objects can be serialized (converted to raw bytes). Processes can spawn subprocesses, or child processes, which are also isolated from each other and the parent. Processes facilitate preemptive multitasking, in which the OS scheduler preempts (suspends execution of each process periodically so that other processes can run) (p.698)

Thread: A thread is an execution unit (the general term for objects that execute code concurrently, each with independent state and call stack) within a process. A process initializes a single thread (the main thread), and can spawn more threads to operate concurrently by calling operating system APIs. All threads within a process share the same memory space, which contains the active Python objects, thereby facilitating easy data sharing among threads. The tradeoff here is data corruption when one or more thread attempt to update the same object concurrently. Threads also enable preemptive multitasking in conjunction with the OS scheduler (p.698).

Types of Concurrent vs. Parallel Programming

Multithreading refers to a type of concurrent programming in which a process spawns and orchestrates multiple threads to complete a task. Multithreading implements preemptive multitasking, meaning the OS scheduler is responsible for switching contexts (i.e., deciding which thread is running). Each thread can be executing its own set of instructions, but the the Global Interpreter Lock (GIL) ensures that only one thread can be running at any given time. Thus, it is impossible to utilize multiple processors with threads in Python. The GIL serves as a mutex (a type of lock that prevents more than one thread from accessing a state) to prevent multiple threads from executing Python bytecode at the same time. This is necessary since CPython (the interpreter that executes the bytecode instructions) does not have thread-safe memory management (2).

Asynchronous programming is another form of concurrent programming available through the asyncio library via async/await syntax. A key difference between asyncio and multithreading is that asyncio implements cooperative multitasking, meaning there is only a single thread and control is passed between coroutines (functions defined with the async def; can suspend itself and resume later). When an await statement is reached, the current coroutine pauses execution, passing control back to the event loop until the current awaited operation returns. While an in-depth discussion of the asyncio event loop is beyond the scope of this article, you can think of the event loop as the manager of all the coroutines, coordinating what task to switch to and when to resume a paused coroutine (3).

Multiprocessing enables us to leverage multiple CPU cores via parallelization, with each core capable of handling its own tasks and processing information independently. This is achieved via the multiprocessing package in Python, and supports spawning multiple processes, circumventing the GIL since it uses subprocesses rather than threads. The key difference is that these subprocesses do not share the same memory space, meaning there is significant overhead cost associated with sharing data between processes.

Race Conditions, & Deadlock

A common pitfall in multithreaded programming is known as a race condition, which occurs when two or more threads attempt to access and modify the same data at the same time and the final result is dependent on the order in which the threads run. For example, say you have two threads (T1 and T2) trying to access your email address in a system. T1 will update your address while T2 is reading the address as it’s been requested. If T1 and T2 are executed simultaneously, T2 could return your old (outdated) email, or the correct (updated) email, depending on which completes first. Deadlock is another concurrent computing phenomenon, which can arise when two or more processes or threads are blocked because each one is waiting for the other to release a resources, bringing progress to a halt.

Synchronization Primitives

To protect against race conditions when employing concurrent programming, we use synchronization primitives, which are effectively mechanisms used to control access to shared resources. While these are similar in both multithreading and asyncio, the synchronization primitives for asyncio are designed for cooperative multitasking, meaning they are not thread-safe and should only be used within the same event loop. Additionally, asyncio primitives do not accept the timeout argument, opting for asyncio.wait_for() to implement timeouts. The multiprocessing module also includes equivalents of all the synchronization primitives from threading, but these are not as necessary in a multiprocess program so they will not be discussed in this article. Furthermore, the official documentation recommends avoiding shared state and using queues or pipes for communication between processes rather than lower level synchronization primitives.

Multithreading

Locks and RLock

A lock is used to allow only one thread to access a resource at a time, such that once the lock is acquired, no other threads can acquire the lock until the lock is released. locks are either locked or unlocked, and can be acquired by calling the Lock.acquire() method. The release() method, is used to unblock execution of other threads when called by the thread holding a locked lock (but will raise a RuntimeError if called on an unlocked lock). locks can be used as a context manager to automate acquisition and releasing of locks as shown below:

import threading 
import time
from concurrent.futures import ThreadPoolExecutor

lock = threading.Lock()
class UserAccount:
    def __init__(self):
        self.name = "John Doe"
        self.email = "John@gmail.com"
        self.account_lock = threading.Lock()
 
    def get_email(self):
        with self.account_lock:   
            return self.email
    
    def set_email(self, email):
        with self.account_lock:
            self.email = email

with ThreadPoolExecutor(max_workers=3) as executor:
    executor.submit(set_email, "Johndoe@gmail.com")
    executor.submit(get_email)
    executor.submit(set_email, "John@gmail.com")

One drawback with the regular lock is that if the same thread attempts to acquire the lock it already holds, a deadlock will occur. This could happen if within your locked function you invoke another function containing the same lock. Enter, the RLock, or reentrant lock. This synchronization primitive allows the same thread to acquire a lock multiple times before releasing it, preventing deadlock in situations where a thread needs to re-enter a locked resource. The tradeoff is increased overhead since the RLock has to keep track of how many times the same thread has acquired a lock, so it should only be used when necessary (4).

Semaphores

The semaphore was invented in the early 1960's by someone every CS student is familiar with: Edsger W. Dijkstra.

A semaphore is a type of atomic counter which guarantees that the OS will not interrupt the thread in the middle of incrementing or decrementing the counter. The internal counter is incremented/decremented with the release() and acquire() methods, respectively. Semaphores are frequently used to protect a resource with limited capacity, such as a connection pool (4). Semaphores are constructed by passing in the max number of concurrent threads acquiring it. Semaphores can be used as context managers, entering with a successful acquire() call and automatically calling release() when exiting the with block.

The example below (adapted from asyncio.Semaphore: Practical Guide with Real-World Use Cases) highlights this use case by utilizing a semaphore to cap the number of queries that can execute concurrently using the max_connections variable.

import threading
from concurrent.futures import ThreadPoolExecutor
import psycopg2
from psycopg2 import pool


class DatabasePool:
    def __init__(self, dsn, max_connections=5):
        self.dsn = dsn
        self.semaphore = threading.Semaphore(max_connections)
        self.pool = None

    def init_pool(self):
        self.pool = psycopg2.pool.ThreadedConnectionPool(
            minconn=1,
            maxconn=5,
            dsn=self.dsn,
        )

    def query(self, sql, *args):
        with self.semaphore:
            conn = self.pool.getconn()
            try:
                with conn.cursor() as cur:
                    cur.execute(sql, args)
                    return cur.fetchall()
            finally:
                self.pool.putconn(conn)

    def close(self):
        self.pool.closeall()


def main():
    db = DatabasePool("postgresql://user:password@localhost/database")
    db.init_pool()

    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = [
            executor.submit(db.query, "SELECT * FROM users WHERE id = %s", i)
            for i in range(1, 11)
        ]
        results = [f.result() for f in futures]

    db.close()


if __name__ == "__main__":
    main()

Events

Events are objects that allow threads to "communicate" via an internal flag that defaults to False, but can be set to True by calling the set() method, or reset to False by calling clear(). Individual threads can wait for the flag using the wait() method, which blocks execution until the flag is set. Flags can be used to coordinate actions across multiple threads, such as signalling state changes, thereby enabling efficient synchronization management. When a new Costco location opens the often go all out, selling limited-time champagne signed by celebrities and serving caviar samples. The example below uses events to simulate the opening of a new Costco location and the commencement of giving out caviar:

import threading
import time
from concurrent.futures import ThreadPoolExecutor

costco_open = threading.Event()
caviar_open = threading.Event()

def serve_customer(customer_data):
    print(f"{customer_data['name']} is waiting for the Costco to open.")

    costco_open.wait()
    print(f"{customer_data['name']} entered Costco")
    if customer_data["type"] == "VIP_EXPERIENCE":
        print(f"{customer_data['name']} is waiting for caviar to be served.")
        caviar_open.wait()
        print(f"{customer_data['name']} is getting their caviar.")

        # Simulate the time taken for eating caviar
        time.sleep(2)

        print(
            f"{customer_data['name']} finished eating and exited the store"
        )
    else:
        # Simulate the time taken for shopping
        time.sleep(2)
        print(f"{customer_data['name']} has exited the store")

customers = [
    {"name": "Customer 1", "type": "REGULAR_SHOP"},
    {"name": "Customer 2", "type": "VIP_EXPERIENCE"},
    {"name": "Customer 3", "type": "REGULAR_SHOP"},
    {"name": "Customer 4", "type": "VIP_EXPERIENCE"},
]

with ThreadPoolExecutor(max_workers=4) as executor:
    for customer_data in customers:
        executor.submit(serve_customer, customer_data)

    print("Costco manager is preparing to open the store.")
    time.sleep(2)
    print("Costco is now open!")
    costco_open.set()  # Signal that the new location is open

    time.sleep(3)
    print("Caviar is now being served!")
    caviar_open.set()

print("All customers have completed their experiences.")

Conditional Waiting

A condition object is built on top of a Lock or RLock and supports additional functionality allowing threads to wait for certain conditions to be met, and signal other threads of condition changes.

Methods associated with Condition objects:

• acquire(): Acquire the underlying lock; must be called before a thread can wait on or signal a condition

• release(): Releases the underlying lock

• wait(timeout=None): Blocks the thread until it’s notified, or a specific timeout occurs. The lock is released before blocking and reacquired upon notification or when timeout expires. Useful when a thread needs to wait for a specific condition to be true before proceeding

• notify(n=1): Wakes up one of the threads waiting for the condition (if any are waiting). Will select one randomly if multiple threads are waiting

• notify_all(): Wakes up all threads waiting for the condition

Condition objects are useful for coordinating across threads and managing the flow of execution in a multithreaded environment. The following example from from Real Python demonstrates how a condition can be used to synchronize access to the shared customer_queue object and signal the teller_thread when a new customer arrives.

import random
import threading
import time
from concurrent.futures import ThreadPoolExecutor

customer_available_condition = threading.Condition()

# Customers waiting to be served by the Teller
customer_queue = []

def now():
    return time.strftime("%H:%M:%S")

def serve_customers():
    while True:
        with customer_available_condition:
            # Wait for a customer to arrive
            while not customer_queue:
                print(f"{now()}: Teller is waiting for a customer.")
                customer_available_condition.wait()

            # Serve the customer
            customer = customer_queue.pop(0)
            print(f"{now()}: Teller is serving {customer}.")

        # Simulate the time taken to serve the customer
        time.sleep(random.randint(1, 5))
        print(f"{now()}: Teller has finished serving {customer}.")

def add_customer_to_queue(name):
    with customer_available_condition:
        print(f"{now()}: {name} has arrived at the bank.")
        customer_queue.append(name)

        customer_available_condition.notify()

customer_names = [
    "Customer 1",
    "Customer 2",
    "Customer 3",
    "Customer 4",
    "Customer 5",
]

with ThreadPoolExecutor(max_workers=6) as executor:
    teller_thread = executor.submit(serve_customers)
    for name in customer_names:
        # Simulate customers arriving at random intervals
        time.sleep(random.randint(1, 3))
        executor.submit(add_customer_to_queue, name)

Barriers

Barriers allow groups of threads to wait for each other before continuing execution. It blocks program execution until a specified number of threads reach the barrier

Barrier takes in one required and two optional arguments:

• parties: the # of threads of the barrier object that the wait() method waits for before proceeding

• action: callable that will be executed by one of the threads when released

• timeout: timeout value for the wait() method

Going back to Costco, let's say that we want to use a barrier to only initiate the Grand Opening (open the flood gates, if you will) once all employees are prepared for the stampede:

import random
import threading
import time
from concurrent.futures import ThreadPoolExecutor

employee_barrier = threading.Barrier(3)

def now():
    return time.strftime("%H:%M:%S")

def prepare_for_work(name):
    print(f"{now()}: {name} is preparing their station.")

    # Simulate the delay to prepare the station
    time.sleep(random.randint(1, 3))
    print(f"{now()}: {name} has finished preparing.")

    # Wait for all employees to finish preparing
    employee_barrier.wait()
    print(f"{now()}: {name} is now ready to serve customers.")

employees = ["Front door person 1", "Front door person 2", "Cashier 1", "Cashier 2", "Caviar sample distributor"]

with ThreadPoolExecutor(max_workers=5) as executor:
    for employee_title in employees:
        executor.submit(prepare_for_work, employee_title)

print(f"{now()}: All employees are ready to serve customers.")

Barriers are useful when multiple threads need to be in sync with each other before proceeding, or when you need to coordinate the start of a sequence across multiple threads (4). For example, you could use a barrier when threads are working concurrently to compute a set of results to ensure that all results are in before proceeding to the next stage of computation.

Asyncio

The asyncio synchronization primitives function very similarly to those of threading, with the goal of coordinating access to shared resources between tasks. This section will serve as a high-level overview, so refer to the official docs for more information!

Lock

Locks are used to guarantee exclusive access to a shared resource. The recommended utilization is an async with statement

lock = asyncio.Lock()

async with lock:
    # access shared state

The acquire() and release()methods are similar to those to threading.Lock:

• acquire()waits until the lock is unlocked, then sets it to locked and returns True. Only one coroutine can proceed at a time when more than one are waiting for the lock to be unlocked. Acquiring a lock is fair, meaning the coroutines proceed in the order they arrived (FIFO)

• release() method resets the lock to unlocked, and raises a RuntimeError if called on an unlocked lock

• The locked() method returns True if the lock is locked

Event

Events are used to notify multiple asyncio tasks that an event has happened. events contain an internal flag that can be set to True with the set() method and reset to False with clear() method. The wait() method blocks execution until flag is set to True. When the set() method is invoked, all tasks waiting on the event will be immediately awakened.

Condition

Conditions combines functionality of events and locks. Conditions can be initialized by passing in an existing lock, otherwise a new lock will be created automatically. Conditions allow coordinating exclusive access to a shared resource between tasks.

Condition methods include:

• The acquire() method acquires the underlying lock, waiting until the lock is unlocked.

• notify(n) and notify_all() wake up n tasks or all tasks that are waiting on the condition object, respectively.  The lock must be acquired first, otherwise a RuntimeError will be raised.

• The locked() method returns True if the underlying lock is currently acquired

• release()releases underlying lock, resulting in a RuntimeError if called on an unlocked lock

• wait() causes a task to wait, blocking execution until notified

• wait_for(predicate)blocks execution until the specified predicate (a callable that is interpreted as a boolean) evaluates toTrue

Semaphore

Manages an internal counter, decremented by each acquire() call and incremented by each release() call. Semaphores are instantiated by passing in a value (default is 1), specifying the number of tasks that can access the protected resource at a time. The counter can never go below zero, so if acquire() is called at 0, it blocks until another task releases it.

Bounded Semaphore

• A bounded semaphore is a version of a semaphore that raises a ValueError if the release() method would cause the internal counter to go above the initial value.

Barrier

Similar to threads, barriers block execution at a point in the code until a specified number of parties reach it, at which point all tasks are unblocked simultaneously.

Summary: When should we use what?

Key point: neither multithreading nor asynchronous programming will ever increase the power in a system due to the cost of task switching and synchronization. As such, good candidates for these techniques are tasks that involve significant downtime waiting for external events (i.e., I/O-bound tasks).

Multithreading is desirable in Python due to preemptive task switching, meaning no additional code is required to context switch. Further, all threads share a state, so no additional overhead is required for threads to access a shared resources such as variables and data structures. However, this is a double-edged sword since multiple threads accessing and shared state can result in race conditions affecting critical sections, and deadlock can arise as a result of protecting such sections. Locks are problematic because it becomes very difficult to reason about your code as more are added (1). Furthermore, locks don't actually "lock" anything– they're just a signal that a thread can check– but if a thread doesn't check to acquire a lock, they can still access the resource.

Between async and multithreading, asyncio may be desirable to multithreading since you no longer have to deal with locks or worry about arbitrary interruptions. Additionally, the cost of switching contexts is very low because it uses generators under the hood to store the state (1). The disadvantage of the asynchronous approach is that you have to explicitly add keywords such as yield or await to pass control back to the scheduler and manage the flow of execution. Furthermore, every single thing you do has to be non-blocking, and it requires all involved libraries to be async-compatible (1).

For CPU-bound tasks, we opt for multiprocessing, since such tasks block the event loop when executed asynchronously, and the Python GIL causes threads to perform worse than sequential code during intense computations. Indeed, Python is very popular in the modern data science community, supporting compute-intensive tools such as Project Jupyter, TensorFlow, PyTorch, and Dask.

Sources:

1. Raymond Hettinger, Keynote on Concurrency, PyBay 2017

2. The Python GIL (Global Interpreter Lock)

3. Python's asyncio: A Hands-On Walkthrough

4. Python Thread Safety: Using a Lock and Other Techniques

5. Ramalho, L. (2022). Fluent Python : clear, concise, and effective programming. O’reilly Media, Inc.

   ‌

Live Demo

Video Demo

If you haven’t read them already, check these out first:

Part 1 – Messaging

Part 2 – User Permissions & Security

Overview

The nice thing about adding file attachments is that the user experience is straightforward. I knew that I wanted to have some sort of button that the user clicks which opens up a file explorer where they can then select the attachment(s) they wish to upload. What didn’t occur to me is the complexity involved in both the frontend to seamlessly display the attachments, and the backend to store the attachments with the messages.

File Storage Best Practices

When I first started this project, I didn’t understand why we can’t simply store the file in the database itself. In short, objects smaller than 256KB are best stored in a database whereas larger objects (>1MB) are best stored in the file system (Gray, 2006). In other words, databases are optimized for structured queries whereas filesystems are optimized for streaming large object. Other reasons why you shouldn’t store files in your database include such as excessive size, high costs, slow backups, and poor performance. Instead, we will upload the file into a cloud object storage system, such as an S3 bucket, from which we can create a key that is stored in the database. We can then use that key to access the object.

I think it helps to break down the process into distinct steps, and then break down the implementation at each step:

1. User uploads file to a message

2. User sends message with file attachment

3. Message displays in chat with attachment (optimistic update)

4. Retrieve pre-signed URL from backend

5. Upload directly to S3

6. Send message via WebSocket

7. Server saves to database

8. Other clients receive the message

Updating the Data Model

In order for this to be possible, some modifications need to be made to the data model. The approach I took was to create a MessageAttachment model that has a many-to-one relationship with the message table (each message attachment relates to a single message).

// MessageAttachment model
export const MessageAttachment = sequelize.define('messageAttachment', {
  id: {
    type: DataTypes.UUID,
    defaultValue: DataTypes.UUIDV4,
    primaryKey: true,
  },
  fileUrl: {
    type: DataTypes.STRING,
    allowNull: false,
  },
  fileName: {
    type: DataTypes.STRING,
    allowNull: false,
  },
  mimeType: {
    type: DataTypes.STRING,
    allowNull: false,
  },
  messageId: {
    type: DataTypes.UUID,
    allowNull: false,
    references: {
      model: Message,
      key: 'id',
    },
  },
});

// Establish relationships for Sequelize
  Message.hasMany(MessageAttachment, {
    foreignKey: 'messageId',
    as: 'attachments',
  });

  MessageAttachment.belongsTo(Message, {
    foreignKey: 'messageId',
    as: 'message',
  });

Next, I have to modify all controllers that query messages to include the attachments. For example, the following controller gets all messages for a channel:

export const getChannelMessages = catchAsync(async (req, res, next) => {
  const channelId = req.params.id;

  const messages = await Message.findAll({
    where: { [Op.and]: [{ channelId }, { parentMessageId: null }] },
    include: [
      {
        model: Channel,
        as: 'Channel',
        attributes: ['channelName', 'id'],
      },
      {
        model: User,
        as: 'Sender',
        attributes: ['firstName', 'lastName', 'avatarUrl'],
      },
      { model: MessageAttachment, as: 'attachments' },
    ],
  });

  res.status(200).json({
    status: 'success',
    results: messages.length,
    data: messages,
  });
});

Note: the where statement includes the condition parentMessageId: null, which is how I filter out thread replies.

Implementing File Attachments

Step 1: The File Upload

Since we need to access the files outside of the attachment button component, and we need our attachments variable to update when changes are made, we can define a state attachments and setAttachments in the parent component and then pass them in to the upload button component. To create the file upload button I used Chakra’s built-in Input element with the type property set to file. Next, I defined a function handleFileChange which creates an array from the selected files and then updates the attachments state. Finally, I added a handleRemoveFile function to allow the user to remove files that they uploaded by mistake. The entire component looks like this:

import { AiOutlinePaperClip } from 'react-icons/ai';
import { Button, Flex, Text, Input, Box } from '@chakra-ui/react';

export default function ChatFileUploadButton({ attachments, setAttachments }) {
  const handleFileChange = (e) => {
    const selectedFiles = Array.from(e.target.files);
    setAttachments((prev) => [...prev, ...selectedFiles]);
    e.target.value = null; // allow re‑uploading same file name
  };

  const handleRemoveFile = (index) => {
    setAttachments((prev) => prev.filter((_, i) => i !== index));
  };

  return (
    <Box w="100%">
      <Flex align="center" gap={2}>
        <Input
          type="file"
          multiple
          display="none"
          id="chat-file-input"
          onChange={handleFileChange}
        />

        <Button
          as="label"
          htmlFor="chat-file-input"
          rounded="full"
          w="1rem"
          marginLeft={2}
          size="sm"
          bg="bg.primaryBtn"
          color="text.primaryBtn"
          _hover={{ bg: 'bg.navHover' }}
        >
          <AiOutlinePaperClip />
        </Button>
      </Flex>

      {attachments.length > 0 && (
        <Flex direction="column" gap={2} mt={2}>
          {attachments.map((file, idx) => (
            <Flex key={idx} align="center" gap={2}>
              <Text fontSize="sm" flex="1">
                {file.name}
              </Text>
              <Button
                fontSize="sm"
                p={0}
                size="2xs"
                onClick={() => handleRemoveFile(idx)}
              >
                x
              </Button>
            </Flex>
          ))}
        </Flex>
      )}
    </Box>
  );
}

Then I can render my upload button from the parent component (the ChatInput where I initialized the state), which will allow me to access the attachments as they are updated by the user interacting with the interface.

Step 2: Message Submission

An uneventful but imperative stage in which the user clicks the send button or presses enter, initiating the message submission process.

Step 3: Optimistic Update

The optimistic updates are very important to the user experience and performance of the chat application. Without optimistic updating, we would essentially have query the messages from the database every time a message is sent and then re-render the messages for the user. With optimistic updating, we can give the illusion of instantaneous communication without having to query the database each time. Additionally, it allows us to populate the chat immediately as opposed to forcing the user to wait until the file finishes processing, which would be a poor overall experience.

const createOptimisticMessage = ({ messageBody, attachments }) => {
    const now = new Date();
    const datestamp = now.toLocaleDateString('en-US', {
      weekday: 'long',
      year: 'numeric',
      month: 'long',
      day: 'numeric',
    });
    const timestamp = now.toLocaleTimeString([], {
      hour: 'numeric',
      minute: '2-digit',
      hour12: true,
    });

    return {
      tempId: uuidv4(),
      messageBody,
      attachments: attachments.map((file) => ({
        name: file.name,
        type: file.type,
        preview: URL.createObjectURL(file),
      })),
      sender: {
        firstName: userData.firstName,
        lastName: userData.lastName,
      },
      timestamp,
      datestamp,
      status: 'sending',
    };
  };

To summarize, the function above takes in a string messageBody and an array attachments, then returns an object with all information necessary to display a ChatMessage. Of note, we are adding two fields: tempId, which will be used to match the optimistic message with the “real” message from the database, and status, which indicates whether a message has successfully been processed by the server.

Now that we have the ability to create an optimistic message, we can invoke the helper function at the beginning of the onSubmit handler:

const onSubmit = handleSubmit(async (data) => {
    const optimisticMsg = createOptimisticMessage({
      messageBody: data.message,
      attachments,
    });
    
    onMessageSent(optimisticMsg);
    // ...
}

Note: the onMessageSent function is a helper function that is passed down from the ChatInterface and is used to update the messages state. It’s purpose is to add the newly created optimistic message and then sort them based on the date and time.

  const handleMessageSent = (msg) => {
    setMessages((prev) => insertAndSortMessages([...prev, msg]));
  };

  const insertAndSortMessages = (messagesArray) => {
      return [...messagesArray].sort(
        (a, b) =>
          new Date(a.createdAt || a.sentAt) - new Date(b.createdAt || b.sentAt)
      );
  };

Now that the chat has been updated with the most recent message, we can continue with the “…” part of the handleSubmit function.

Step 4: Retrieve pre-signed URL from backend

We do not want to send the entire file via WebSockets for several reasons, including but not limited to:

• It would block other messages while the file is being transmitted

• Poor recovery, meaning dropped connection requires the process to restart, whereas HTTP-based uploads can be chunked/resumed

• Long uploads can trigger timeouts

• Adds significant overhead when encoding binary as base64

Moreover, if you were paying attention earlier you would recall that we shouldn’t be storing files in the database in the first place, so we don’t really have a reason to transmit them in the first place!

Step 4a: Processing Attachments

The better approach is to delegate the file upload process to a helper function that takes in both files and messageId, loads the files into our object storage system, and then returns the data that we want to store in the database. There is one small problem with this approach, however: the backend has the credentials to our S3 bucket, and we want to keep them out of the frontend to avoid exposing them.

The solution here is to generate pre-signed, authenticated URLs generated by AWS. First, we need to create a controller function that will create a unique key for each file, then return an uploadUrl (the URL used for the POST request) and a fileUrl (the location where the uploaded file will be stored).

export const generateUploadUrls = catchAsync(async (req, res, next) => {
  const { files } = req.body; // Array of {name, mimeType}

  if (!files || !Array.isArray(files))
    return next(new AppError('Files array is required', 400));

  const presignedUrls = await Promise.all(
    files.map(async (file) => {
      const ext = path.extname(file.name) || '';
      const key = `messageFiles/\({uuidv4()}\){ext}`;

      const command = new PutObjectCommand({
        Bucket: process.env.AWS_S3_BUCKET_NAME,
        Key: key,
        ContentType: file.mimeType,
      });

      // Generate presigned URL (valid for 5 minutes)
      const uploadUrl = await getSignedUrl(s3, command, { expiresIn: 300 });

      // The final public URL after upload
      const fileUrl = `https://\({process.env.AWS_S3_BUCKET_NAME}.s3.\){process.env.AWS_REGION}.amazonaws.com/${key}`;

      return {
        uploadUrl, // Use this to upload
        fileUrl, // This is the final URL to save in DB
        fileName: file.name,
        mimeType: file.mimeType,
      };
    }),
  );

  res.status(200).json({ status: 'success', data: presignedUrls });
});

Step 4b: Retrieve pre-signed URLs

Now we can define a helper function to retrieve those URLs by making a POST request from the frontend, passing in the necessary data in the request body:

const getPresignedUrls = async (files) => {
  const fileMetadata = files.map((file) => ({
    name: file.name,
    mimeType: file.type,
  }));

  const res = await axios.post(
    `${import.meta.env.VITE_DEV_API_BASE_URL}uploads/generateUploadUrls`,
    {
      files: fileMetadata,
    },
    { withCredentials: true }
  );
  return res.data.data;
};

Step 5: Upload directly to S3

We will also want a helper function to handle the upload itself. While I originally had both functions combined into one, I decided to separate them out into distinct helper functions to abide by the single responsibility principle. It also makes it easier to debug since each HTTP request is made in its own function.

const uploadToS3 = async (file, uploadUrl) => {
  const blob =
    file instanceof File ? file : await fetch(file).then((r) => r.blob());

  await axios.put(uploadUrl, blob, {
    headers: {
      'Content-Type': file.type,
    },
  });
};

Now we can put it all together into our main uploadAttachments function (the one that will be invoked when the user submits the message):

export const uploadAttachments = async (files, messageId) => {
  try {
    // Get presigned URLs from backend
    const presignedData = await getPresignedUrls(files);

    if (!presignedData || !Array.isArray(presignedData)) {
      throw new Error('Invalid presigned URLs response');
    }
    // Upload each file to S3 using presigned URLs
    await Promise.all(
      files.map((file, index) =>
        uploadToS3(file, presignedData[index].uploadUrl)
      )
    );

    // Return the file records to save in DB
    return presignedData.map((data) => ({
      fileName: data.fileName,
      messageId,
      fileUrl: data.fileUrl,
      mimeType: data.mimeType,
    }));
  } catch (error) {
    console.error('Error uploading attachments:', error);
    if (error.response) {
      console.error('Server response:', error.response.data);
    }
    throw error;
  }
};

Step 6: Send message via WebSocket

Because we processed the files into an array of objects containing metadata about the attachments (instead of the files themselves), we can modify the sendMessage function to include that array in the messageContent object that is sent over the socket.

  const sendMessage = async ({ messageBody, attachments = [], tempId }) => {
    if (!userSocket?.connected) return; // Ensure socket is connected

    let uploadedAttachments = [];
    if (attachments.length) {
      try {
        uploadedAttachments = await uploadAttachments(attachments, null);
      } catch (err) {
        console.error('Failed to upload attachments: ', err);
        return;
      }
    }

    const now = new Date();
    const datestamp = now.toLocaleDateString('en-US', {
      weekday: 'long',
      year: 'numeric',
      month: 'long',
      day: 'numeric',
    });
    const timestamp = now.toLocaleTimeString([], {
      hour: 'numeric',
      minute: '2-digit',
      hour12: true,
    });

    const messageContent = {
      messageBody,
      attachments: uploadedAttachments,
      sender: { firstName: userData.firstName, lastName: userData.lastName },
      timestamp,
      datestamp,
    };

    const messageData = {
      messageContent: messageBody,
      senderId: userData.id,
      tempId,
      type: chatMode === 'ch' ? 'channel' : 'direct',
    };

    // Conditional logic to ensure message includes necessary information
    if (chatMode === 'ch') messageData.channelId = channel?.id;
    if (chatMode === 'dm') {
      messageData.receiverId = directMessage?.id;
      messageData.roomId = channel?.id;
    }

    if (chatMode === 'thread') {
      messageData.parentMessageId = thread?.parentMessage?.messageId;
      userSocket.emit('send-thread-message', messageContent, messageData);
    } else {
      userSocket.emit('send-message', messageContent, messageData);
    }

    return messageContent;
  };

Step 7: Server saves to database

After the sendMessage function completes, our socket connection in the backend will receive the data since it is listening for the send-message event. Upon receiving the data from the frontend, we need to store the attachment metadata and ensure that it is properly associated with the message it came attached to. I accomplished this using a helper function saveAttachmentRecords which takes in an array of attachment metadata attachments and the messageId:

export const saveAttachmentRecords = async (attachments, messageId) => {
  // Save to your Attachment model
  const results = [];

  for (const att of attachments) {
    const record = await MessageAttachment.create({
      fileName: att.fileName,
      fileUrl: att.fileUrl,
      mimeType: att.mimeType,
      messageId,
    });

    results.push(record);
  }

  return results;
};

Step 7a: Save attachments to database

The first addition we need to make to the send-message handler function is to call the helper function above, which is what will actually create the relationship between message and attachment in the database:

 if (messageContent.attachments?.length) {
    const attachments = await saveAttachmentRecords(
       messageContent.attachments,
       messageId,
     );
   }

Step 7b: Build confirmed message with the real messageId

Remember the tempId and status fields we added earlier? This is where they resurface. Now that the files have been processed and the messages are stored in the database, we create the confirmedMessage, which replaces the optimistic message from Step 3.

 // Build confirmed message with real ID
 const confirmedMessage = {
    ...messageContent,
    id: messageId,
    tempId: messageData.tempId,
    status: 'sent',
};

 // Send confirmation to sender
 socket.emit('message-confirmed', confirmedMessage);

Now we can add an event listener to the frontend to replace the optimistic message with our confirmed message:

const handleMessageConfirmed = (msg) => {
    // Replace optimistic message with confirmed one
    setMessages((prev) =>
      prev.map((m) => (m.tempId === msg.tempId ? msg : m))
    );
};

Step 8: Other clients receive the message

Finally, we broadcast the confirmedMessage to the appropriate room (using the channelId if it’s a channel message and roomId for direct messages) by emitting the receive-message event.

// Broadcast to others in the room
if (messageData.type === 'channel' && messageData.channelId) {
    socket.broadcast
      .to(messageData.channelId)
      .emit('receive-message', confirmedMessage);
  } else if (messageData.type === 'direct' && messageData.roomId) {
      socket.broadcast
      .to(messageData.roomId)
      .emit('receive-message', confirmedMessage);
  }

This event is then received by our socket connection in the frontend and the involved clients’ interfaces are updated with the message (and potentially attachment):

const handleReceiveMessage = (msg) => {
  setMessages((prev) => insertAndSortMessages([...prev, msg]));
};

Final Thoughts

This process is confusing and took me a LONG time to fully wrap my head around. It involved making several changes and additions to the existing architecture. I also implemented it entirely differently at first and then had to completely revise my approach when I realized it wasn’t working. My first attempt involved sending the entire file over the socket connection (encoded in Base64). I eventually got it to work sending small files, but I quickly ran into issues trying to scale because the file uploads kept blocking the connection (because as discussed earlier, Web Sockets are not meant for sending files). The second time I blundered was when I added my AWS credentials into the frontend environment variables, not realizing that would expose them to anyone who decided to look in the developer tools.

This project taught me about choosing the right tool for the job, and the importance of understanding the benefits and drawbacks of different systems, services, data structures, and algorithms. In other words, just because you can do something doesn’t mean you should do that thing. Instead, seek to understand if the system or tool you’re using is designed to complete the task at hand. If you’re not sure, it might be worth doing some research to deepen your understanding and explore alternatives before proceeding. As much as it pains me to admit it, I do see the benefit of Leetcode style assessments that train your problem solving ability by teaching you to recognize patterns and then find the best solution using the tools (data structures and algorithms) at your disposal. There are often many ways to solve such a problem, but the “optimal” solution is not always obvious without comparing different approaches.

Live Demo

Video Demo

In the first part of this series, I outlined the basic project set up and reviewed how I implemented real-time messaging in my application, so give that a read if you haven’t already!

User Permissions

Roles and Role-Specific Actions

In order to enforce user permissions, we must first create a hierarchy of roles which dictate what actions a user can or cannot do within the application. In ChattyCat, I decided on the following roles: owner, admin, superuser, member.

Refer to the table below which contains the basic actions available and to whom they are available:

At this time, any individual can create an account and then create their own organization, which would automatically make them the owner. Of note, the owner of the organization is the only member who has the ability to assign roles to members in their organization– all users are initially assigned the member role by default.

Enforcing User Permissions

Now the question is: how do we actually enforce the roles and role-specific actions from above? At a high level we need to:

1. Keep track of who has what role (within the organization).

2. Verify that the user making the request has the required permissions to execute an action.

Well, couldn’t we simply send the role with the rest of the user data when the request is made from the frontend? It seems logical, and that’s what my more naive self thought when first posed with this challenge. However, there is a significant security concern: what if someone were to make a request directly to the backend and simply modify the role parameter? Indeed, a better solution involves implementing middleware in our server to “protect” certain endpoints. Before going into detail on that middleware, we first have to understand how roles are managed in our data model.

Roles in the Data Model

Initially when building ChattyCat, it was designed such that each user could only be part of a single organization, and “role” was a property of the user. As the application developed, I realized this would require a new user to be created for the same person who wants to be part of multiple organizations, so I revised the relationship. Given the many-to-many cardinality between users and organizations, this presented a new challenge: a user could have different roles in each organization they are a member of!

This required the creation of a through table (also commonly referred to as junction or bridge table), which effectively creates two one-to-many relationships and stores extra details about the specific relationship. In this example, we create the UserOrganization table to keep track of the user’s role and joinedAt date. The result is a new table that has a one-to-many relationship from the User table on the userId key, and a one-to-many relationship from the Organization table on the organizationId key.

To create the table in Sequelize, you define a new model which contains foreign keys for the tables you wish to join, and then you can specify the additional properties.

import { DataTypes } from 'sequelize';

import sequelize from '../utils/database.js';
import User from './userModel.js';
import Organization from './channelModel.js';

const UserOrganization = sequelize.define(
  'UserOrganization',
  {
    userId: {
      type: DataTypes.UUID,
      references: {
        model: User,
        key: 'id',
      },
    },
    organizationId: {
      type: DataTypes.UUID,
      references: {
        model: Organization,
        key: 'id',
      },
    },
    role: {
      type: DataTypes.ENUM('owner', 'admin', 'member', 'superuser'),
      allowNull: false,
      defaultValue: 'member',
    },
    joinedAt: {
      type: DataTypes.DATE,
      defaultValue: DataTypes.NOW,
    },
  },
  {
    timestamps: false,
  },
);

export default UserOrganization;

Next, you establish the many-to-many relationship and specify the through table:

  Organization.belongsToMany(User, { through: UserOrganization, as: 'Users' });
  User.belongsToMany(Organization, {
    through: UserOrganization,
    as: 'Organizations',
  });

The code above is establishes the following relationships depicted as an entity relationship diagram:

Now to set the roles we can use instance methods provided by Sequelize, passing in values for columns in the through table (read about associations in the documentation if interested):

export const assignRole = catchAsync(async (req, res, next) => {
  // This function will only be available to the owner
  const { userId, role } = req.body;
  const orgId = req.headers['x-organization-id'];

  const org = await Organization.findByPk(orgId);
  const user = await User.findByPk(userId);

  // addUser will create association if it doesn't exist, or update the through table if it does.
  await org.addUser(user, { through: { role } });

  res.status(200).json({
    status: 'success',
    message: `User \({user.firstName} \){user.lastName} was assigned role: \({role} in organization \){org.organizationName}`,
  });
});

Writing the Middleware

The next step is to ensure that only the owner of the organization can successfully execute the assignRole function. To accomplish this, I created the requireOrgRole middleware, which takes in a list of allowed roles, looks up the user’s role in the organization, then verifies whether or not the user’s role falls within the accepted roles. Additionally, it adds the organizationId and the user’s role to the request for easy access by subsequent middleware.

export const requireOrgRole = (allowedRoles = []) => {
  return async (req, res, next) => {
    const orgId = req.headers['x-organization-id'];

    if (!orgId) {
      return next(new AppError('requireOrgRole: Missing organization ID', 400));
    }

    // Lookup the user's role in the org
    const membership = await UserOrganization.findOne({
      where: {
        userId: req.user.id,
        organizationId: orgId,
      },
    });
    
    // Throw an error if the user is not part of the organization
    if (!membership) {
      return next(
        new AppError('User is not a member of this organization', 403),
      );
    }

    if (!allowedRoles.includes(membership.role)) {
      return next(new AppError('Insufficient permissions', 403));
    }

    // Attach org and role info to request if needed later
    req.organizationId = orgId;
    req.orgRole = membership.role;

    next();
  };
};

Since requests pass through middleware in the order they appear, we can restrict the assignRole functionality to only users who have role owner in our router:

router.use(requireOrgRole(['owner']));
router.post('/assignRole', assignRole);

This pattern can be repeated to enforce all the role-specific actions defined previously. For example, in our channel router adding the following ensures that only those with role admin, owner, or superuser can create or delete channels:

router.use(requireOrgRole(['admin', 'owner', 'superuser']));
router.post('/', createChannel);
router.route('/:id').delete(deleteChannel);

Security

Protecting User Data

Password Hashing & Salting

Without users, an application is nothing, so it is imperative to protect the user’s sensitive information at all costs. First and foremost, user passwords should never be stored in the database itself. Instead, upon registration a hashed password will be created and stored in the database. Hashing refers to a one-way function that takes in some string and produces a fixed output. Salts are commonly added to hashing functions to add random data, thus increasing security and reducing risk of identifying duplicate or common passwords (Source). After registering, future login attempts will take in the input password and generate a new hash, then compare it to the existing one in the database to authenticate the user. This can be completed using packages such as bcrypt combined with the beforeSave hook provided by Sequelize.

User.beforeSave(async (user, options) => {
  if (user.changed('password')) {
    const hashedPassword = await bcrypt.hash(user.password, 12);
    user.password = hashedPassword;

    if (!user.isNewRecord) user.passwordChangedAt = Date.now() - 1000;
  }
  user.passwordConfirm = undefined;
});

JSON Web Tokens for Authentication

Additionally, we can keep track of a timestamp passwordChangedAt to ensure that the JSON Web Token issued to the user is current. To make sense of this, let’s review how authentication takes place in ChattyCat.

JSON Web Tokens (JWTs) provide a a secure method for sending information between a client and a server, consisting of JSON data secured with a cryptographic signature. There are three components of a JWT:

Source: JSON Web Token (JWT), Geeksforgeeks.

The header contains information about the token, such as the algorithm used for signing. The payload contains the data that is being sent. The signature verifies the integrity of the token, and is generated using the header, payload, and a secret key. We can work with JWTs by installing the jsonwebtoken library (npm i jsonwebtoken), which exposes methods for signing and verifying tokens that are sent and received. For example, the following helper function is used to sign a JWT:

const signToken = (id) =>
  jwt.sign({ id }, process.env.JWT_SECRET, {
    expiresIn: process.env.JWT_EXPIRES_IN,
  });

We can then create middleware to send the signed token to the client as a cookie, which is how the user will remain logged in:

const createSendToken = (user, statusCode, req, res) => {
  const token = signToken(user.id);
  const cookieOptions = {
    expires: new Date(
      Date.now() + process.env.JWT_COOKIE_EXPIRES_IN * 24 * 60 * 60 * 1000,
    ),
    httpOnly: true,
    secure: process.env.NODE_ENV === 'production',
    sameSite: process.env.NODE_ENV === 'production' ? 'None' : 'Lax',
  };

  res.cookie('jwt', token, cookieOptions);

  user.password = undefined;

  res.status(statusCode).json({
    status: 'success',
    token,
    data: {
      user,
    },
  });
};

Finally, we can create our login function which brings everything together, first checking that the entered password matches the hashed password, then sending a signed JWT:

export const login = catchAsync(async (req, res, next) => {
  const { email, password } = req.body;
  // 1) Check if email and password are populated
  if (!email || !password)
    return next(new AppError('Please provide email and password!', 400));

  // 2) Check if user exists && password is valid
  const user = await User.findOne({ where: { email } });
  if (!user || !(await user.correctPassword(password, user.password))) {
    return next(new AppError('Incorrect email or password', 401));
  }
  // 3) Send token to client
  createSendToken(user, 200, req, res);
});

Now since we sent the JWT as a cookie to the frontend, future requests coming from the frontend will have the JWT attached. We can use this to create an additional middleware that will be used for all actions where we want to ensure a valid user is making the request. This is also where we will confirm that the user hasn’t changed their password since the JWT was issued, which prevents unauthorized users from using stale logins.

export const protect = catchAsync(async (req, res, next) => {
  // 1) Get token and check if it's there
  const token = req.cookies.jwt;

  if (!token)
    return next(
      new AppError('You are not logged in! Please log in to get access.', 401),
    );

  // 2) Verify token
  const decoded = await promisify(jwt.verify)(token, process.env.JWT_SECRET);

  // 3) Check if user still exists
  const currentUser = await User.findByPk(decoded.id);
  if (!currentUser)
    return next(
      new AppError('The user belonging to this token no longer exists', 401),
    );

  // 4) Check if user changed password after token was issued
  if (currentUser.changedPasswordAfter(decoded.iat)) {
    return next(
      new AppError('User recently changed password! Please log in again.', 401),
    );
  }

  // Finally grant access to protected route and add user to the request
  req.user = currentUser;
  next();
});

Security Middleware

In addition to the methods above to protect passwords and provide secure authentication, there are other common attacks that we need to defend against. I’ll preface this section by saying it is NOT comprehensive by any means and I am not an expert in cybersecurity. I’m simply sharing what I’ve learned about what to watch out for and how I handled them in this application.

Brute Force Attacks

These attacks involve continuous attempts to login with common passwords. While this is partially mitigated by the delay provided when using bcrypt, other strategies such as rate limiting (setting a limit for the number of requests that can come from one user or IP in a given time period) or implementing a maximum number of login attempts can provide additional protection.

I used express-rate-limit package to implement this in my application:

// Limit requests from same IP
const limiter = rateLimit({
  max: 100,
  windowMs: 60 * 60 * 1000,
  message: 'Too many requests from this IP, please try again in an hour!',
});
app.use('/api', limiter);

Cross-Site Scripting (XSS) Attacks

An XSS attack involves an attacker tricking the target site into executing malicious code. For example, an attacker might send a malicious code within the request’s URL parameter that gets embedded into the page that is then return to the browser. There are two primary methods to protect against XSS attacks:

1. Output Encoding: escaping potentially harmful characters. Modern frontend frameworks such as React, Vue, and Angular provide output encoding by default when rendering variables.

2. Sanitization: removing unsafe features from a string of HTML, such as a <script> tag or inline event handler. It’s worth noting that sanitization is something that’s only really needed when rendering raw HTML, which does not occur when using normal JSX components and standard HTML elements; raw HTML is only rendered when using the dangerouslySetInnerHTML element in React. Thus, we don’t need to do anything here, but you ever render raw HTML in your application, Mozilla recommends using a trusted third party library such as DOMPurify for sanitization.

Furthermore, packages such as Helmet can be added to the Express middleware stack to set special headers which further secure your application. Finally, use a Content Security Policy as a failsafe to control which resources the browser is allowed to execute.

Denial of Service (DoS) Attacks

DoS attacks involve overwhelming the server with a large number of requests or requests that require significant resources to complete. Rate limiting will provide protection against such attacks, but we can further defend against such attacks by restricting the size of the payload attached to the request body. The middleware below enforces limits on the size of the JSON payload and URL-encoded form data respectively.

app.use(express.json({ limit: '10kb' }));
app.use(express.urlencoded({ extended: true, limit: '10kb' }));

Reseting Passwords via Email

Since the passwords are never stored anywhere in plaintext and hashing is one-way (meaning you cannot “unhash” a password), a forgotten password can never be recovered. Given how many times I have had to reset passwords through email, it never occurred to me how complicated this process would be to implement, so I’ll do my best to walk through it. For this to work, the user model must include properties to keep track of a reset token and an associated expiration date (I used fields passwordResetToken and passwordResetExpires). When a user is unable to login, the following should occur:

1. User clicks “forgot password” button and is directed to a form where they can enter the email associated with their account.

2. Upon submission, a request to the backend is made and invokes a forgotPassword function.

3. The forgotPassword function checks if there is a user matching the provided email and then generates a password reset token and expiration time.

4. User receives an email with a link for them to set a new password.

5. Once new password is set, the user is logged in and the information is updated in the database.

Creating the Form (ForgotPassword.jsx)

The form is pretty basic, containing a single input element and a submit button, which invokes a helper function sendResetEmail that makes an API call to the backend with the email provided (not going into depth about the API call as that’s something I discussed at length in this article).

The forgotPassword Function

This function should do three things in order: find the user, generate the reset token, and then send the email with the password reset link. Finding the user is straightforward, but to create the reset token I added an instance method that utilizes the crypto module (built-in to Node.js) to generate and random token. This function also hashes the token, sets the token as the user’s passwordResetToken, and finally sets the passwordResetExpires :

User.prototype.createPasswordResetToken = function () {
  const resetToken = crypto.randomBytes(32).toString('hex');

  this.passwordResetToken = crypto
    .createHash('sha256')
    .update(resetToken)
    .digest('hex');

  this.passwordResetExpires = Date.now() + 10 * 60 * 1000;

  return resetToken;
};

Once we generate that token, we then have to call user.save to update the database (note that I disabled validators to prevent them from throwing an error):

  // 2) Generate Reset Token
  const resetToken = user.createPasswordResetToken();
  await user.save({ validate: false });

Next, we construct the reset URL:

const resetURL = `\({req.protocol}://\){process.env.FRONTEND_URL}/resetPassword/${resetToken}`;

• req.protocol refers to HTTP/HTTPS, whatever protocol the sender is using

• process.env.FRONTEND_URL is self-explanatory, but this allows us to dynamically send the email based on the environment

• resetPassword indicates that this is the route on the frontend we are targeting (i.e., in our React Router we render a resetPassword page when that route is hit).

• resetToken is the token we just generated and will be appended to the URL so that it’s available to React Router as a parameter (by using the useParams hook).

Finally, we send the email to the user, which I handled by creating an Email class containing a sendPasswordReset method (will elaborate on this in the next section):

await new Email(user, resetURL).sendPasswordReset();

Putting it all together we get the following:

export const forgotPassword = catchAsync(async (req, res, next) => {
  // 1) Find user by POSTed email
  const user = await User.findOne({ where: { email: req.body.email } });

  if (!user)
    return next(
      new AppError('There is no user associated with that email address.', 404),
    );

  // 2) Generate Reset Token
  const resetToken = user.createPasswordResetToken();
  await user.save({ validate: false });

  // 3) Send to user's email
  try {
    const resetURL = `\({req.protocol}://\){process.env.FRONTEND_URL}/resetPassword/${resetToken}`;
    await new Email(user, resetURL).sendPasswordReset();

    res.status(200).json({
      status: 'success',
      message: 'Token sent to email!',
    });
  } catch (err) {
    console.error(err);
    user.passwordResetToken = undefined;
    user.passwordResetExpires = undefined;
    await user.save({ validate: false });

    return next(
      new AppError('There was an error sending the email. Try again later!'),
      500,
    );
  }
});

Sending Emails

I used Nodemailer (Node.js library for sending emails) and Mailgun (email service) to send emails. To do so, you must create a transporter with Nodemailer, which is the object that handles connecting to your email service. Once created, it can then be used to send an email by calling the sendMail function. Since email services are typically not free (or you don’t want to use up the free tier testing), I utilized Ethereal in development to emulate an email service. For more information, the Nodemailer documentation is very helpful!

The Email class constructor takes in a user and a url, then initializes the variables that we will need to pass in when creating an email.

class Email {
  constructor(user, url) {
    this.to = user.email;
    this.firstName = user.firstName || '';
    this.url = url;
    this.from = 'ChattyCat <no-reply@chattycat.dev>';
  }
// ...
}

The generic newTransport method is used to create a transport using either Mailgun or Ethereal, depending on the environment.

  newTransport() {
    if (process.env.NODE_ENV === 'production') {
      const mailgunOptions = {
        auth: {
          api_key: process.env.MAILGUN_API_KEY,
          domain: process.env.MAILGUN_DOMAIN,
        },
      };
      return nodemailer.createTransport(mg(mailgunOptions));
    }

    // Development
    return nodemailer.createTransport({
      host: 'smtp.ethereal.email',
      port: 587,
      auth: {
        user: testAccount.user,
        pass: testAccount.pass,
      },
    });

The generic send method takes in a subject and a message, constructs a mailOptions object, and then invokes the Nodemailer sendMail function with those options.

  async send(subject, message) {
    const transporter = await this.newTransport();

    const mailOptions = {
      from: this.from,
      to: this.to,
      subject,
      text: message,
    };

    const info = await transporter.sendMail(mailOptions);
  }

Finally, the sendPasswordReset method is just a send method where the subject and message are pre-populated. This pattern allows us to keep all the logic for sending emails contained to the Email module.

  async sendPasswordReset() {
    await this.send(
      'Your password reset token (valid for 10 minutes)',
      `Forgot your password? Navigate to ${this.url}`,
    );
  }

The end result:

import nodemailer from 'nodemailer';
import mg from 'nodemailer-mailgun-transport';

const testAccount = await nodemailer.createTestAccount();

class Email {
  constructor(user, url) {
    this.to = user.email;
    this.firstName = user.firstName || '';
    this.url = url;
    this.from = 'ChattyCat <no-reply@chattycat.dev>';
  }

  newTransport() {
    if (process.env.NODE_ENV === 'production') {
      const mailgunOptions = {
        auth: {
          api_key: process.env.MAILGUN_API_KEY,
          domain: process.env.MAILGUN_DOMAIN,
        },
      };
      return nodemailer.createTransport(mg(mailgunOptions));
    }

    // Development
    return nodemailer.createTransport({
      host: 'smtp.ethereal.email',
      port: 587,
      auth: {
        user: testAccount.user,
        pass: testAccount.pass,
      },
    });
  }

  async send(subject, message) {
    const transporter = await this.newTransport();

    const mailOptions = {
      from: this.from,
      to: this.to,
      subject,
      text: message,
    };

    const info = await transporter.sendMail(mailOptions);
  }

  async sendPasswordReset() {
    await this.send(
      'Your password reset token (valid for 10 minutes)',
      `Forgot your password? Navigate to ${this.url}`,
    );
  }
}

export default Email;

Preventing Unwanted Messages

The steps outlined above ensure that user data is secure, all requests are coming from valid users, users can reset their password if they forget it or have a security concern, and old JWTs cannot be used to spoof a user login after their password is changed. However, this is not sufficient to prevent valid users from sending messages or files to channels they should not have access to.

Protecting users from unauthorized messages takes place in the backend when the send-message event is received by the socket. To implement this, I created a helper function validateUserPermissions to ensure that the user sending the message is a member of the channel they are sending the message to:

const validateUserPermissions = (userId, channelId) => {
  if (!channelId) return true;
  const channels = userChannelMap.data.get(userId);
  return channels.includes(channelId);
};

Note that we need the check at the start so that messages without a channelId (which are direct messages) will not get blocked erroneously. At this time, all users are capable of sending messages to each other within an organization, and the only way to be added to an organization is by an administrator, so we only need to verify the channel permissions at this time.

You may also be wondering about the userChannelMap in this function. Originally, this method involved making a database query to get a list of all the channels the user is a part of. However, given this function is invoked every time a message is sent, it would become very resource intensive. To address this, I implemented the UserChannelMap data structure to keep track of all users and their associated channels. The map is initially populated with all users and their respective channels when the server spins up, and is updated whenever a request to add or remove somebody from a channel is processed. While maintaining this cache does increase the memory requirements, I believe that the trade-off is worth it compared to a database query whenever a message is sent.

Final Thoughts

Most of the concepts covered in this article fall under the umbrella of “things I didn’t know I had to worry about before this project.” 🤷‍♂️

Stay tuned for Part 3 of this series, where I will be discussing file attachments!

Live Demo

Video Demo

Project Overview

Given the complexity of this project, I will be publishing 3 separate posts that build off one another. The first in the series (this post) will focus on messaging. The next post will focus on user permissions and security, and the final installment on attachment handling.

My most recent project is easily the most complex challenge I’ve tackled to date, combining everything I’ve learned about RESTful APIs, frontend development, user experience, and security. ChattyCat is a web application where users can send messages to channels and other users, reply in message threads, upload profile pictures, and even securely reset their passwords via email. Users can send messages to any other user within their organization, but can only send messages to channels that they have been added to by a superuser or administrator.

Backend

The backend for this application is written with Node.js and Express, using a NeonDB PostgreSQL instance to store user and message data. Profile picture uploads are supported via Multer for processing the file and the AWS S3 client for loading the image into an S3 bucket. The data models and relationships are constructed using Sequelize V6.

Frontend

The frontend is created using React, Vite. I used React Router (version 7) to implement routing, and Chakra V3 for some components and additional styling.

Implementing Web Sockets with Socket.io

The first challenge this project presented was utilizing web sockets to facilitate real-time communication.

What is a WebSocket?

A WebSocket is used to establish real-time, bidirectional communication between a client and a server (Mozilla Docs). This differs from the HTTP request-response model since the TCP connection stays open until it is explicitly closed. Additionally, both client and server can send messages along the connection without making new requests, reducing overhead costs and making it ideal for web chat applications (source).

Setting up Socket.io

For this project I used Socket.IO (version 4) to establish the channel between server and client. Once the connection is established, the server and client interact by emitting and listening for events, and then responding accordingly.

If you want to follow the Socket.io tutorial, you can do that here, although my implementation is slightly different so I want to outline the high-level steps and design patterns I followed rather than focus on the specific details.

Socket.IO Lifecycle

The Socket.IO lifecycle can be summarized in 3 main steps (Socket.IO docs):

1. IO server starts and begins listening for new socket connections

2. Client creates a socket, which initiates a WebSocket handshake to connect to the IO server

3. IO server receives the connection

Step 1: Server-side

To create the connection from the server side we need to to first initialize a Socket.IO Server (imported from the socket.io package), which is created by passing in an HTTP server. In the example below, app is an Express instance, so calling app.listen returns an HTTP server. Thus, io is an instance of the Socket.IO server where we add a listener for the “connection” event for incoming sockets, which will fire each time a new connection is established. Each connection represents an individual client’s session, meaning all socket.on listeners are bound to that specific client socket.

import { Server } from 'socket.io';

import app from './app.js';

const PORT = process.env.PORT || 3000;

const server = app.listen(PORT, () => {
  console.log(`App is running on port ${PORT}...`);
  console.log(`mode is ${process.env.NODE_ENV}`);
});

// Step 1: IO Server starts and begins listening for new socket connections
const io = new Server(server, {
  cors: {
    origin: [
      'http://localhost:5173',
      'http://127.0.0.1:5173',
      // Make sure to include your development and production domains here
],
    credentials: true,
  },
});

io.on('connection', (socket) => {
  console.log('a user connected');
});

Step 2: Client-side

To establish the connection from the client-side we will use the Socket.IO client library (npm i socket.io-client). I decided to move this code into a utility function that “promisifies” the socket creation to allow for asynchronous execution. The purpose of the createConnection() function is to initialize the connection, which is why I chose to contain the logic of socket event handlers (e.g., socket.on(‘receive-message’)) within the component(s) that use them.

import { io } from 'socket.io-client';

export const createConnection = () => {
  return new Promise((resolve, reject) => {
    // Step 2: Client creates socket
    const socket = io(import.meta.env.VITE_SERVER_URL, {
      withCredentials: true,
    });

    socket.on('connect', () => {
      console.log('[CLIENT] Connected with socket ID:', socket.id);
      resolve(socket); 
    });

    socket.on('connect_error', (err) => {
      console.error('Socket connect_error:', err);
      reject(err);
    });

  });
};

Note: we set withCredntials: true since it is a cross-origin request and we want the browser to send credentials (cookies, authorization headers, TLS certificates) with the request.

Step 3: Bidirectional Communication

Following the lifecycle steps above, let’s say we started our IO server and then call createConnection() from our frontend application. At this point the connection event listener will fire, and log a the message “a user connected” to the console. This is not particularly useful just yet, but I think it’s important to conceptualize what is happening and where before we can start to develop with the WebSockets. What we just established is the ability to programmatically interface with either part (server-side or client-side) of the same persistent connection. While this may be obvious to some, it was not to me, and I’m sure at least one other person out there has struggled to wrap their head around this concept.

Implementing the Messaging Functionality

Emitting events

The client communicates with the server by emitting (socket.emit) and listening (socket.on) for events which accepts an object as data. The name of the event is the first argument, and will be used by the server to identify the emitted event. For example, you can send a message from an input form:

  const socket = await createConnection(); // Just pretend this is valid for now

  const form = document.getElementById('form');
  const input = document.getElementById('input');
  const messages = document.getElementById('messages'); // Where we will display messages

  form.addEventListener('submit', (e) => {
      e.preventDefault();
      if (input.value) {
        socket.emit('send-message', input.value);
        input.value = '';
      }
  });

The above snippet emits a send-message event along with the value of the input, which now needs to be received by the server.

To respond, we can add a send message listener by modifying our backend code where we initialized the IO server and listened for the connection event. Inside the listener, the first argument is the name of the event (should match the one we emitted above), and the second argument is a callback function that takes in the object we passed in as our data:

io.on('connection', (socket) => {
  console.log('a user connected');

  socket.on('send-message', (msg) => {
    console.log(`[SERVER] message received from client \({socket.id}: \){msg}`);
    socket.emit('receive-message', msg); // Send back to client so we can display in frontend
  })    
});

At this point, the user has sent a message (presumably “hello world!”), upon which a send-message event was emitted by the client socket. The IO server then executes the functionality defined in the send-message listener, logging both the client’s socket id and message to the console, and then emits a receive-message event.

To finish this sequence of events and display the message in the chat, we need to define a receive-message listener, which will render the message data in the UI. Recall earlier I mentioned how I used createConnection() ONLY for establishing the initial connection. This is because I required the use of React hooks such as useEffect to manage sockets and clean up listeners upon unmount or socket change. To achieve this functionality, we retrieve a userSocket from our UserContext (refer to the GitHub if you want to see how that is initialized), and then we add the receive-message listener function which updates a messages state array containing all messages.

After the IO server fires the receive-message event, our client-side socket is set up to update the messages array, which will subsequently re-render the component due to the state update. So all that’s left is to display the messages, which can be completed by iterating through our messages state array and rendering a new message element for each item in the array!

import { useState, useEffect } from 'react';

export default function ChatMessages() {
  const { messages, setMessages } = useState([]);
  const { socketReady, userSocket } = useContext(UserContext);

  useEffect(() => {
    if (!userSocket) return;

    const handleReceiveMessage = (msg) => {
      setMessages((prev) => [...prev, msg]);
    };

    userSocket.on('receive-message', handleReceiveMessage);

    return () => {
      userSocket.off('receive-message', handleReceiveMessage);
    };
  }, [userSocket, setMessages]);
  return (
    <>
      {messages.forEach((msg, idx) => (
        <p key={idx}>{msg}</p>
      ))}
    </>
  );
}

Learning through Development

The more I develop “real” applications, the more I appreciate the difficulty and art of writing clean code that is both reusable and maintainable. A project can get out of hand very quickly, making it difficult locate the component, function, or sometimes even the file you are searching for. In order to avoid this common pitfall, you need both logical project structuring and strong pattern recognition.

Creating Reusable React Components

My focus for this project was to improve the organization and iteratively refactor to minimize duplicate code. Tech debt racks up quickly when you start duplicating code, often causing simple tasks to become burdensome when you have to make changes to multiple files/components instead of just one.

The first component that I refactored was my chat layout, which initially stored and displayed all messages for a chat and the input box. I knew this component would be reused at least once because I would require different chat interfaces for direct messaging vs. sending messages to a channel, and I wanted to avoid maintaining two separate React components.

First, I identified which components could be separated functionally, which resulted in a split between the ChatInput (sending messages) and MessageLayout (displaying messages). However, if I were to separate these two components, I would then have to lift the messages state outside of these components since it needs to be accessed by both. Ultimately, I lifted up the messages state into the parent components where I return the ChatInterface. After this initial separation of ChatInput and MessageLayout, I further deconstructed MessageLayout into two components: DividerText and ChatMessage. The reason for this decision was based on more on reusability than any logical separation, since each component could be useful independently for future design. For example, DividerText is now used for both separating messages and differentiating the start of a message thread from its replies. Furthermore, it can be customized by passing a style prop, which would not have been possible before without duplicating many lines of code.

The image below provides a visual representation of how I reduced my original ChatInterface component into groups of smaller components which are more reusable and maintainable. This offers the added benefit of improving the project’s file structure since I can now move each component to its own aptly-named file, which is a significant improvement to lumping everything into the more abstract parent ChatInterface component.

Throughout this project I improved my ability to identify where a component can be refactored to make the codebase more maintainable. Previously, my components would continue to grow as I added functionality, increasing the number of dependencies, making it very difficult to debug, and resulting in lots of duplicated code. This time I was guided by the Single Responsibility Principle, which posits that a component should only have one responsibility. Ultimately this required a shift in my coding mentality, causing me to frequently evaluate whether the changes or additions I’m making would be better suited in their own component or helper function. While it can feel bad in the moment to remove code from a function or component that “works,” it often saves time in the long run and leads to a significantly more manageable codebase. I also noticed that it makes it easier to return to the project after not looking at it for a time and quickly understand the purpose of each component.

Commitment Issues…

Something else I improved on this project is strategically utilizing Git to “tell the story” of the project’s development. In the past I would often forget to commit for long periods of time, which made it difficult to create meaningful commit messages since the commit would contain so many changes. I tried to make each commit address a specific change or addition, which resulted in many more commits but also a more robust version history. While I’m still working on the art of deliberate commits, a good formula to follow is a short summary of the change (≤50 characters), an empty line, then a detailed summary focusing on the intent and approach (wrapped at 72 characters). For more guidance, I’d recommend checking out this blog post by Peter Hutterer or this FreeCodeCamp blog.

Final Thoughts

I think I am finally starting to understand the “engineering” part of software engineering. The ability to write code is simply the prerequisite for being a software engineer. Most of the real work is the planning, design, and scaling a project using all the tools (i.e., languages and frameworks) at your disposal. For example, when implementing threaded messages I had two approaches in mind:

1. Create a new table threads with a 1:many relationship with messages? (each thread can have many messages; each message can have at most 1 thread)?

2. Add a parentMessageId column to the message table, which indicates that a message belongs to a thread.

In the first approach, it allows me to leave the message table as is, and I could easily distinguish between a “parent” message and a “thread” message; however, I would have to create a whole new table. The latter approach involves modifying the existing message table and requires adding some additional logic to distinguish between different message types, BUT I wouldn’t have to add any new tables or create new relationships.

Ultimately I decided to go with approach number two because it kept the data model leaner and contained all messages to a single table, but perhaps I could have achieved the same results with the first approach. The choices made throughout development will impact the performance and sustainability of the project, and being able to identify the best (or at least better) approach is integral to successful software engineering.

Deployed Project

Overview

The MERN stack (MongoDB, Express.js, React, and Node.js) consists of all the tools and frameworks you need to deploy a fully functional, modern web application by only writing code in a single language (JavaScript). By using Express and Node you can construct RESTful API endpoints to interact with your MongoDB database. Meanwhile, React is used to build a responsive front-end user experience. Throughout this blog post I will discuss my experience process of building my first complete application and deploying it to the web.

Project Planning

I started the project by first defining the requirements and technical details. I started by mapping out the user experience, answering questions such as:

• What should the user see when first visiting the page?

• What actions should the user be able to take?

• What function does this application serve for the user?

With the answers to the questions above guiding me, I moved onto the high-level technical requirements:

• React frontend using Chakra UI component library for styling.

• Frontend should implement routing for different pages

• RESTful API back end, completely separate from frontend.

• Secure authorization and authentication using JavaScript web tokens.

• Mobile responsive.

The next step was to draw a wireframe, outlining the components needed for each page of the website. You can see an example of one of the pages below. If you’d like to see the whole wireframe on Excalidraw, click here.

Building the Frontend

Project Set Up

Once the initial project planning was complete, it’s time to set up your project structure. Best practice is to initialize a source folder as a git repository, then create folders for the frontend and backend to help prevent dependency/environment clashes and make it easier to deploy them independently. Getting started on a full stack project can be incredibly daunting, but I’ve found that sometimes you just have to start somewhere. For me, it made the most sense to start with the frontend because I already had the basic outline. Additionally, it’s more motivating when I can visualize the progress for what I’m working on.

Within your frontend folder, use Vite to initialize the React project by running the command npm create vite@latest ./ and answering the prompts (don’t forget to install Vite if you don’t already have it). Make sure to select React as the framework, then choose your preferred variant (I used JavaScript). Upon completion, you will have a basic React app that can be run using the command npm run dev.

Setting Environment Variables with Vite

The first thing I do is create a .env file to store the variables necessary to execute the code in my development environment. When using Vite, environment variables must follow the naming convention of starting with “VITE_” and containing all capital letters. For example: VITE_API_KEY=abc123 is a valid environment variable, but API_KEY=abc123 is not. To access these variables within your code, utilize the syntax import.meta.env.VITE_API_KEY.

Getting Started with React

Now that we have a way to see the changes made in real time, it’s time to start building the home page.

Using my wireframe as a reference, I was quickly able to identify the key components I would need to build:

1. Navigation bar

2. Navigation bar buttons

3. Search bar

4. Movie/Show Carousel

5. Movie/Show Carousel Card

Implementing Chakra (v3)

Chakra UI provides a library of sleek, customizable components, which significantly cuts down on the time spent building the individual pieces of the user interface. It also allows provides a built-in ColorModeButton component for toggling between light and dark mode, which is an essential feature in any modern application.

I used Chakra v3 for this project, which can be installed by running npm i @chakra-ui/react @emotion/react. Next add the code snippets with npx @chakra-ui/cli snippet add, which generates the provider component (among others) which is essential for the next step. I’d also recommend installing React Slick (npm i react-slick), which is used for later implementing the Slider portion of the carousel.

Now all you have to do to start developing with Chakra is wrap your App component in the Provider that was generated by Chakra

import { Provider } from "@/components/ui/provider";
import App from "./App.jsx";

createRoot(document.getElementById("root")).render(
   <Provider>
      <App />
   </Provider>
);

Just ONE more thing before we can actually get started… Given that I wanted to be able to implement routing in my web app, we first have to set that up with React Router (I used v7 for this project).

Setting up React Router

Install React Router by running npm install react-router. Once installed, you need to wrap your App component inside the BrowserRouter

import { BrowserRouter } from "react-router-dom";      
import App from "./App.jsx";

createRoot(document.getElementById("root")).render(        
    <BrowserRouter>
      <Provider>
         <App />
      </Provider>
    </BrowserRouter>
);

Inside your App component return a <Routes> element which wraps your individual <Route> elements (the only difference here is the “s”, confusing, I know). Each <Route> element specifies a path and the element it should render.

import { Routes, Route } from "react-router";
import RootLayout from "./pages/Root";
import HomePage from "./pages/HomePage";

import "./App.css";

function App() {
  return (
    <Routes>
      <Route path="/" element={<RootLayout />}>
        <Route index element={<HomePage />} />
      </Route>
    </Routes>
  );
}

You can also specify query parameters on the path, for example: <Route path="/movie/:mediaId" element={<MovieDetails />} />. Then to access the parameter, React Router provides the usePrams hook, which exposes an object of all specified parameters:

import { useParams } from "react-router-dom";
let { mediaId } = useParams();

Note that I created an element <RootLayout> which wraps my other routes. The use of the <Outlet> is what allows my navigation bar to persist on all pages without having to import it and render it in each individual component. Here you can also see how to use Chakra elements by importing them from the library we installed earlier.

import { Outlet } from "react-router-dom";
import { Box } from "@chakra-ui/react";
import MainNavigation from "../components/MainNavigation";

export default function RootLayout() {
  return (
    <Box as="main" minHeight="100vh">
      <MainNavigation />
        <main>
          <Outlet />
        </main>
    </Box>
  );
}

With all that out of the way, we can start building the components that the user will interact with. I won’t go through each component individually as this isn’t intended to serve as a “how to frontend”, but you’re welcome to check out the source code from the GitHub repo if interested.

Getting Data from The Movie Database (TMDB)

With the components built using some sample movie data, the next step is to integrate live data from TMDB using their API. Once you setting up a developer account and generate an API key, make sure to add it to your .env file with an appropriate name. Next, we will make some test requests using Postman by making a GET request and providing our API token in the “Authorization” section of the request (Navigate to Authorization and select “Bearer Token” from the Auth Type drop down menu).

To make your life even easier, set up a new environment within Postman and create variables for Base Url and your API token, which can be accessed using the double bracket notation shown above.

Sending the request above will yield a JSON object response containing the popular TV shows, which we can then reference when writing building the frontend components.

{
    "page": 1,
    "results": [
        {
            "adult": false,
            "backdrop_path": "/2746UvsbkZINd873Yd3o3TxOwCP.jpg",
            "genre_ids": [
                9648,
                18
            ],
            "id": 200875,
            "origin_country": [
                "US"
            ],
            "original_language": "en",
            "original_name": "IT: Welcome to Derry",
            "overview": "In 1962, a couple with their son move to Derry, Maine just as a young boy disappears. With their arrival, very bad things begin to happen in the town.",
            "popularity": 370.308,
            "poster_path": "/nyy3BITeIjviv6PFIXtqvc8i6xi.jpg",
            "first_air_date": "2025-10-26",
            "name": "IT: Welcome to Derry",
            "vote_average": 8.008,
            "vote_count": 123
        },
       ...]

For example, when building our carousel cards we now know the names of the properties to use to access the title, ratings, number of votes, and even poster image.

Making API Calls with Axios

In a new folder (I like to create a utils folder, then a js folder nested within that), we will write the code to handle our asynchronous API calls using Axios and get the data into our main application. Start by installing Axios (npm i axios), then try and replicate the HTTP request we just made with Postman (a GET request to the https://api.themoviedb.org/3/tv/popular endpoint). To do this, we will use the async/await pattern and then a try/catch block to handle errors. If all goes well, this function will return an object containing the results array we saw in the JSON response above. If not, it will throw an error that will be propagated to where the function is invoked.

import axios from "axios";

export const fetchData = async () => {
  try {
    const popularShowRes = await axios({
      method: "GET",
      url: `${import.meta.env.VITE_TMDB_API_BASE_URL}tv/popular`,
      headers: {
        Authorization: `Bearer ${import.meta.env.VITE_TMDB_API_TOKEN}`,
        "Content-Type": "application/json",
      },
    });

    if (popularShowRes.status !== 200)
      throw new Error("Failed to fetch popular shows");

    return {
      popularShowData: popularShowRes.data.results,
    };
  } catch (err) {
    console.error("Error fetching data: ", err);
    // Handle errors here or throw them to be handled where the function is called
    throw err;
  }
};

Working with the API Data in React

Now that we have a way to receive the data, we need a way to elegantly work with it in React. To avoid an infinite loop of loading data into a state variable, then the component re-rendering due to a state change, we use React’s useEffect hook, which allows us to specify an array of variables that will trigger a the effect on change. The implementation below also employs a common React pattern of having state variables for loading and error, which are manipulated inside useEffect and allow us to provide an enhanced user experience by indicating the status of the request. In this case, we do not need to add any variables to the array, which indicates that the useEffect code should only be invoked when the component is rendered.

import { useState, useEffect, useContext } from "react";

import { fetchData, fetchMediaDetails } from "../utils/js/apiCalls";

export default function HomePage(){
    const [data, setData] = useState(null);  
    const [loading, setLoading] = useState(true);
    const [error, setError] = useState(null);  

    useEffect(() => {
        fetchData()
          .then((data) => {
            setData(data);
            setLoading(false);
          })
          .catch((err) => {
            setError(err);
            setLoading(false);
          });
      }, []);
    }  

  // Example of displaying conditional messages to user while loading or in case of error
  if (loading) return <p>Loading...</p>;
  if (error) return <p>Error loading data!</p>;

With the useEffect in place, we can now access the data from our API call as if it were a normal state variable, since we will not reach the return unless the data loads without issue. For example, we could create a carousel using the popular show data by destructuring the object returned from our utility function, and then passing it to our custom carousel component!

const { popularShowData } = data || {};
return ( 
  <ChakraCarousel
        carouselData={popularShowCarouselData}
        title="Popular Shows"
        type="tv"
  />);

Building the Backend

Planning for the Backend

Similarly to the frontend, we start by outlining the building blocks of the backend:

• MongoDB entities (what documents we will need and how we will organize them)

• HTTP endpoints for our API (i.e., GET, POST, PATCH, and DELETE)

For the MongoDB entities, I knew I wanted to have a User document that contains both personal information and the data for each user’s wishlist, watched, and favorites. Thus, each user should have the following properties:

• First Name

• Last Name

• Email

• Password (hashed)

• Movie Wish List

• Movie Favorites List

• Movie Watched List

• TV Wish List

• TV Favorites List

• TV Watched List

For the HTTP End Points, I organized these by verb, including what data the request should include for POST and PATCH requests:

• GET

  • /users (get all users)

  • /users/userId (get one user by id)

  • /users/:userId/watched (get specified list for a specific user)

• POST

  • /users/signup

    • includes: name, email, password, and password confirm

  • /users/login

    • includes: includes email and password

  • /users/me/watched (add an item to a specific list for current user)

    • includes: the id of the item to delete

• PATCH

  • /users/:userId (update specific user data)

    • includes: new key-value pairs for data to update, such as first name, last name, or email (but not password)

  • /users/updateMe (update data for the currently logged in user)

    • includes: new key-value pairs for data to update, such as first name, last name, or email (but not password)

• DELETE

  • /users/:userId (delete a specific user)

  • /users/deleteMe (delete currently logged in user)

  • /users/me/watched (delete an item from a specific list for current user)

    • includes: the id of the item to delete

I decided to only implement endpoints for updating the list for the current user since the functionality would require a user to be logged in. Thus, we could add the user to the request using additional middleware before it reaches the endpoint where the controller is invoked, since middleware is called in the order it is declared in an express application.

Setting up the Backend

Start by initializing a new project in your backend directory (remember how we created two folders at the beginning, one for frontend and one for backend?) with npm init. The only important configuration when setting this up is to enter "module” when the set-up prompts you to select a type (the default is “commonJS”), which will allow you to use ES6 import syntax. If you miss that step, you can always edit it directly in your package.json file! Once you finish initializing your project, it’s time to start installing dependencies:

• MongoDB

• Mongoose

• Express

You can install all 3 at once with the command npm i mongodb mongoose express

Setting up our App with Express

To initialize our application with Express we will set up a test route that will handle GET requests and send a simple response of “API is running”

import express from 'express';

const app = express();

// Test Route
app.get('/', (req, res) => {
  res.send('API is running...');
});

// Default error handling
app.get('/{*any}', (req, res) => {
  res.status(404).json({
    status: 'fail',
    message: `Can't find ${req.originalUrl} on this server!`,
  });
});

export default app;

Connecting to MongoDB

To connect to MongoDB, we will use the mongoose.connect function, which takes in a connection string and a configuration object.

First, you’ll need to register for an Atlas account. Once you answer the registration prompts, you’ll have the option to deploy a free cluster and then create a user. After creating your user, add the password to your backend environment variables since this is how we will connect to MongoDB.

To get your connection string, navigate to your Atlas dashboard and click “Connect,” go to “Drivers”, then select “view full code sample.”

Add the connection string to your environment variables, then in server.js (the main entry point to our backend) replace the password with your MongoDB password by using dotenv.

import dotenv from 'dotenv';
dotenv.config({ path: './config.env' });

const DB = process.env.DATABASE.replace(
  '<db_password>',
  process.env.MONGODB_PASSWORD,
);

Next, we’ll use mongoose to connect to the database:

import mongoose from 'mongoose';

mongoose.connect(DB, {}).then(() => {
  console.log('DB connection successful!');
});

Finally, we’ll start our server by having our App listen for requests on port 3000:

const port = process.env.PORT || 3000;

const server = app.listen(port, () => {
  console.log(`App running on port ${port}...`);
  console.log('mode is', process.env.NODE_ENV);
});

Modelling Data with Mongoose

Starting with the MongoDB entities, since we want to implement individualized profiles where users can keep track of their own lists, we will need a User document. To keep things simple, in addition to the basic user information, we will initialize the lists that will later hold the id’s of TV shows and movies as empty arrays within our user schema. Using Mongoose, we define the properties each user will have by creating a schema, and then export it as a model.

import mongoose from 'mongoose';
const userSchema = new mongoose.Schema(
  {
    firstName: {
      type: String,
      required: [true, 'User must have a first name'],
      trim: true,
    },
    lastName: {
      type: String,
      required: [true, 'User must have a last name'],
      trim: true,
    ...
    tvWishlist: Array,
    tvWatchlist: Array,
    movieWishlist: Array,
    movieWatchlist: Array,
    tvFavoritelist: Array,
    movieFavoriteList: Array,
  },
);

const User = mongoose.model('User', userSchema);

export default User;

Next we need to build the user controller, which is where we will create the functions that will be invoked when our server handles requests from different endpoints. This will be more clear once we mount the routers, so for now just pretend that it makes sense. For example, if we want an endpoint that returns a list of all users, we write the following getAllUsers function:

import User from '../models/userModel.js';
import catchAsync from '../utils/catchAsync.js';

/* catchAsync is a helper function that avoids having to
   put all async functions in try/catch blocks */

export const getAllUsers = () =>
  catchAsync(async (req, res, next) => {
    const doc = await User.find();

    res.status(200).json({
      status: 'success',
      results: doc.length,
      data: {
        data: doc,
      },
    });
  });

Using the user model we created with Mongoose, we can use the find() method, which will return a list with all of the documents from that model.

Next, we need some way for the server to “know” when to invoke the functions that we are defining in our controller files. To do this, we will mount routers, then define the functions we want to execute when our server receives a request. At the top of the app (below where we initialize the app with express), import the userRouter (which we are about to build). At the bottom of the app (right below the test route), add app.use(‘/api/v1/users’, userRouter) which will direct our server to the router we are about to define when requests are made to the /api/v1/users endpoint.

const app = express();
import userRouter from './routes/userRoutes.js';

// Test Route
app.get('/', (req, res) => {
  res.send('API is running...');
});

app.use('/api/v1/users', userRouter);

To define the routes, create a new routes folder, then within that a new file userRoutes.js. Initialize the router by calling express.Router(), then use the route function to define the path. Then we can define the desired behaviors for different HTTP verbs using chaining.

import express from 'express';
import {
  getAllUsers,
} from '../controllers/userController.js';

const router = express.Router();

router.route('/').get(getAllUsers);

export default router;

For example, this will invoke the getAllUsers method we defined previously when the server receives a GET request at the api/v1/users/ endpoint.

Now all you have to do is start the server locally, and it will respond to requests as they come in!

Using the process outlined above, I developed a REST API to handle user signup, login/logout, updating personal information, and even updating the movie/TV show lists. Since all of this data is stored in MongoDB and I use Mongoose to create and update the documents, any changes made to the data will persist in the cloud, meaning requests made from anywhere (including the frontend of the application) to the API are capable of sending and receiving current data about the users.

A Note on Cookie Settings:

When working with HTTPS, make sure to set the secure and sameSite cookie options! I set these conditionally based on the NODE_ENV since the local development server is not secure.

secure: process.env.NODE_ENV === 'production',
sameSite: process.env.NODE_ENV === 'production' ? 'None' : 'Lax',

Deploying the Frontend

To deploy the frontend I used Netlify. After signing up, navigate to “Add new project” > “Import an existing project” > “GitHub”, then select the repository that contains your application. During the set up, specify frontend as the base directory and npm run build as the build command.

Note: you may also have to set the “Publish directory” to frontend/dist since this is where Vite stores the production ready code.

Next, paste in your environment variables from the .env file (the button is right above the “Deploy” button), and then Netlify will generate a url where you can view your live application!

Deploying the Backend

I used Render to deploy the backend, which I discussed in more depth in a previous blog. The process is similar to the above (using GitHub to link the repository), but the root directory will be backend, the build command will be npm install, and the start command NODE_ENV=production node server.js. You will also have to specify your environment variables, which can be pasted from your backend .env file. Depending on how your project is set up (i.e., if you have conditional code that depends on the environment), you would set the NODE_ENV to “production” in Render. At this point, Render will build and deploy your code and you will have a fully deployed web application!

Connecting the Frontend and Backend

Since the two parts of the application operate entirely independently, how do they communicate with one another? At a high level, our backend works by creating an HTTP server object that is configured to receive TCP connections on a specified port and IP address. Since we didn’t bind the app to a specific IP address, it will run on all interfaces available, including http://localhost or http://127.0.0.1, which act as virtual servers for testing our web application.

When building network requests, best practice is to use an environment variable inside a template string rather than hard-coding the hostname. For example: `${import.meta.env.VITE_DEV_API_BASE_URL}api/v1/users/me` (where VITE_DEV_API_BASE_URL= http://localhost:3000/ in our local development environment). Then, all we have to do is update that variable to integrate the two parts of the application. Once the backend is deployed, Render will generate a url ending in “onrender.com,” that can be used to update the base url environment variable in your frontend (on Netlify). Once complete, all the network requests coming from frontend will now be directed to the deployed backend!

Concluding Remarks

This is easily my largest software development project to date and it really pushed me to go out of my comfort zone and learn about things that I never would have even considered (the cookie settings, for example). This endeavor represents a milestone in bridging the gap between writing code in a local development environment (i.e., my laptop) and producing an application that can be used by anyone with access to the internet.

While I learned a lot, there are still parts of the code that I don’t fully understand– I know enough to make it work, but not enough to explain what’s happening behind the scenes, or explore the why. With that said, appreciating the gaps in our knowledge and and acknowledging how much we don’t know is a crucial step in the learning process.

I DEFINITELY underestimated how much work is involved in taking a project that runs on my local machine and making it accessible from the web.

In this article I will walk through the steps and tools involved in taking an app from your local machine and deploying it to Render. I will not, however, go into specifics of the technologies utilized or building the application.

Link to GitHub Repo

Deploying to Render

Prerequisites:

In order to deploy your app to render, you must first have a project in GitHub.

Deploying my app to Render was a relatively painless process.

1. Click the “Add new” button in the top right corner.

2. Click “Web Service”.

3. Sign into GitHub, then select the Repository containing the project.

4. Fill out the fields

   1. Note: make sure the “Start Command” field is correct. It is set to $ node app.js by default, but I had to change it to $ node server.js since I use that as my main entry point.

5. Select “Instance Type” (I used the “Free” one since it is for learning purposes).

6. Copy and paste in your environment variables from your .env file.

7. Click “Deploy Web Service” and let Render work it’s magic 🙂

Additionally, make sure that the Auto-Deploy setting is set to “After CI Checks Pass” from the default “On Commit” if you plan to integrate CI/CD (which I walk through how to do using GitHub Actions later on).

Writing Unit Tests with Jest and SuperTest

Figuring out how to get Jest and SuperTest working was overwhelming due to the amount of information that exists about these, so I’ll try to explain the steps that I did to get it working as simply as possible, but first, what are Jest and SuperTest?

Jest: A framework used for testing JavaScript code. Basically you define conditions that indicate whether or not a function or component is working as intended, and Jest will execute your tests so you can see what is or is not behaving as intended.

SuperTest: A Node.js library used for testing APIs by simulating requests and asserting responses.

Preparing Your Environment

1. Install Jest and Supertest using npm

   npm i jest --save-dev and npm i supertest --save-dev

2. In your package.json file, add the following to your scripts: “test”: “jest”. Now you can execute all your testing via Jest simply by calling npm test

3. Inside your project root, create a folder tests. Since my application involves a REST API, the examples will be organized by controller. What’s important to note here is that each of these files will be treated as one Test Suite by Jest (Test Suites are the organizational unit that contain the unit tests).

4. Create the file to contain the tests with the convention fileName.test.js. For example, tourController.test.js in my application.

5. At the top of the file, require request from SuperTest, and app from the entry point of your application (recall that I said server.js was the entry point for my application)

    const request = require('supertest');
    const app = require('../server');
   

Writing Unit Tests

Recall that to thoroughly test our application, we will create unit tests using Jest, and then utilize functionality from the SuperTest library to handle the HTTP requests.

What is Unit Testing?

Unit testing involves testing the smallest possible functional unit of code. In the case of this demo app are the route handlers, which are essentially middleware that are executed when an HTTP request is made to the route specific by the handler.

Your First Unit Test with Jest

While there is an abundance of information and documentation for Jest, I will try to distill down the most crucial information needed to get to get Jest working with your application, and then as you develop your testing suites refer to the documentation to troubleshoot specific issues.

Within your testing directory, create a file sum.js with the following code:

// sum.js
function sum(a, b) {
  return a + b;
}
module.exports = sum;

Next, create a sum.test.js file in the same directory:

// sum.test.js
const sum = require('./sum');

// Demo test
test('adds 1 + 2 to equal 3', () => {
  expect(sum(1, 2)).toBe(3);
});

Finally, from the root of the project, call npm test (the script we defined in “Preparing Your Environment”).

his will do is invoke Jest, which will locate your test files (the ones with the .test extension), and then it will execute all the tests specified within the Test Suite. In order to demonstrate what the different components of the function above do, I changed the expect(sum(1, 2)).toBe(3) to expect(sum(1, 2)).toBe(4) and put a screenshot of the results below.

When the test fails, you can see the test description (what we input as the first argument into the test function) is displayed in red below the failed test, along with the actual received and expected values. This is the default behavior of Jest, but if you’d like to view the content in the describe, it, and test blocks regardless of pass/fail status then execute the command npm test -- --verbose

Side note on it vs. test: both of these functions essentially do the same thing in Jest, it’s primarily a matter of readability. Moving forward I will use it, in which I describe the expected functionality of the test (i.e., what it should do).

Understanding the Basic Components of a Jest test

• Test description: This is the first argument entered into a the test or it function and should be used to describe the expected result of the test.

• Expect: Statements that are evaluated and determine whether a test passes or fails. Typically in the format of expect(value).[operator](comparator).

  • In the example above, the value we are expecting is the result of calling sum(1, 2), then we call the toBe operator and pass in the comparator value of 4, which checks if the result of sum(1,2) evaluates to 4.

  • This is a very simple example, so please refer to the Jest documentation for more information

Testing your API Endpoint with Jest + SuperTest

Now that we’ve reviewed the basic anatomy of a Jest test, let’s bring in the SuperTest request functionality.

Essentially what this function will do is asynchronously make an HTTP request to our application. Using async/await, we store the results in a variable res, then use chaining to specify the the request type and other details, such as what data we will send (in the event of a POST request).

// userController.test.js
const request = require('supertest');
const app = require('../server'); // Main entry point to your app

describe('POST /api/v1/users/login', () => {
  const userData = {
    email: 'name@example.io',
    password: 'test1234',
  };

    it('should authenticate the user properly', async () => {
    const res = await request(app).post('/api/v1/users/login').send(userData);

    expect(res.status).toBe(200);
    expect(res.body.toHaveProperty('authToken'));
  });

 // Additional tests here
});

Three things I want to point out about the code snippet above:

1. describe is used here to create a logical grouping of related tests. In the example above, I created a block that would run several tests on the specified endpoint.

2. We pass in app to the request function call. In order for this to work, your entry point module file must export the app (module.exports = app at the end of server.js in my case, where app is required at the top of server.js)

3. In order to get all my testing working, I had to wrap the code which initialized my server inside a condition that checks if we are in a testing environment:

if (process.env.NODE_ENV !== 'test') {
  const server = app.listen(port, () => {
    console.log(`App running on port ${port}...`);
    console.log('mode is', process.env.NODE_ENV);
  });

Integrating GitHub Actions

Now that we’ve deployed the app to Render and have some unit tests in place, it’s time to utilize GitHub Actions for our Continuous Integration and Continuous Deployment (CI/CD).

What are GitHub Actions?

GitHub Actions is a service that enables us to automate the build, test, and deployment pipeline for our application. The basic building block of GitHub Actions is a workflow, which will run one or more jobs when triggered by certain events in the repository (such as push or pull requests).

I’d recommend following the steps outlined in the official documentation to get started with GitHub Actions. Once you have you have completed the steps outlined in the Quickstart guide, you can simply modify the YAML file to build and test your project by running npm test.

jobs:
  run-unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: '24' # Specify the node version of your project
      - run: npm ci
      - run: npm test

Setting Environment Variables in GitHub Actions

This is the step that caused me the most trouble so I will outline how I resolved it. When building an app, there are almost always going to be details that cannot be exposed (and thus published) to a public repository such as GitHub. These include credentials for services, API secrets, or any other configuration details that shouldn’t be common knowledge. These details are typically saved in a .env file locally, and then injected into the code using a library such as Dotenv.

In order to set the environment variables for your workflow we will use secrets in GitHub Actions:

1. Navigate to your GitHub repository, then go to Settings > Secrets and variables > Actions > Repository secrets.

2. For each variable in your .env file, add it as a secret by clicking “New repository secret,” then filling in the Name (as it appears in your .env file) and Secret fields.

3. In your GitHub Actions workflow, define your environment variables using the following format for each variable defined in your .env file.

env:
  DATABASE: ${{secrets.DATABASE}}
  DATABASE_PASSWORD: ${{secrets.DATABASE_PASSWORD}}

By using the ${{secrets.VARIABLE_NAME}} syntax we are accessing the secrets context made available to us via GitHub Actions.

Once you have set all necessary environment variables, your app run as intended, using the unit tests we defined earlier as the condition for whether or not the code passes the CI test. Additionally, because of the change we made to the Auto-Deploy setting at the beginning, Render will automatically use this workflow to only re-deploy the application when the continuous integration testing passes. Congratulations! Your app is now live with CI/CD 😄

Final Thoughts

I know that each and every one of these steps deserve it’s own whole article, but for the sake of brevity, learning, and sanity, I tried to include only the information that was absolutely essential. From my experience, one of the most daunting (but also exciting) parts about coding is opening up documentation and realizing just how much you don’t know.

As corny as it sounds, I consider myself a lifelong student. There will always be more to learn and better ways to do things. In fact, there are likely one (or more) better ways to accomplish exactly what I just did in this article. However, striving to make everything as “perfect” as possible from the start can impede progress. Let best practices guide you, and ask yourself often: “how can I improve this?”