Part 3 — Architecture Decisions and Engineering Obstacles

As the RxNorm integration framework evolved, the project gradually shifted from being a simple API experiment into a full-scale data engineering challenge. One of the most important lessons learned during this phase was that architecture decisions matter just as much as programming itself. At small scale, many technical approaches appear viable. However, once systems begin processing thousands of API calls, large volumes of healthcare terminology data, and highly variable response structures, weak architectural decisions quickly become operational bottlenecks.

One of the earliest architecture decisions involved determining how to organize the pipeline itself. Initially, it was tempting to create one large script responsible for every task in the process. While this approach may work for smaller projects, it becomes extremely difficult to debug, maintain, or scale. Instead, I transitioned toward a modular pipeline architecture where each major process operated independently. Separate stages handled raw API collection, normalization, therapeutic classification mapping, NDC expansion, QA validation, and SQL exports.

This modular architecture provided several major advantages. First, it simplified troubleshooting because individual stages could be tested independently. Second, it improved scalability by allowing certain portions of the process to be optimized without impacting the entire pipeline. Third, it created a more maintainable framework that could evolve as additional APIs, datasets, or business requirements were introduced.

Another major architectural decision involved schema normalization. One of the most difficult aspects of working with healthcare APIs is that response structures are not always consistent. Similar fields may appear under different naming conventions depending on the endpoint being queried. Nested structures may also vary dramatically between requests. Because of this, the pipeline required a normalization layer capable of mapping inconsistent structures into standardized analytical schemas.

This normalization process became essential for downstream analytics. Without standardization, it would have been nearly impossible to reliably merge therapeutic classes, medication concepts, and NDC relationships across multiple datasets. Building a consistent schema architecture transformed the raw API responses into something usable for enterprise reporting and future AI initiatives.

Performance engineering also became a major obstacle throughout the project. Healthcare APIs frequently enforce request limits, timeout restrictions, and throttling policies designed to prevent excessive traffic. As the pipeline scaled, I encountered issues involving HTTP 429 rate-limit responses, connection timeouts, and inconsistent response durations. These challenges required implementing retry frameworks, adaptive delays, caching strategies, and incremental processing logic.

One particularly important engineering lesson involved understanding that scalability is not solely about speed. A scalable system must also be resilient. This meant designing the framework to recover gracefully from failures rather than terminating entirely when an endpoint returned an unexpected response. Retry logic, status tracking, error categorization, and logging mechanisms became critical components of the architecture.

Another obstacle involved merging multiple healthcare terminology systems together. RxNorm, RxClass, and other related datasets often describe medication relationships differently depending on the endpoint or clinical context. A single RxCUI might connect to multiple therapeutic classes, ingredient structures, or NDC representations simultaneously. Managing these many-to-many relationships required careful relational design and thoughtful merge strategies.

The expansion of NDC normalization introduced additional engineering complexity. Different systems may store NDC identifiers as 9-digit, 10-digit, or 11-digit structures depending on formatting standards and business requirements. Building a framework capable of expanding and standardizing those identifiers became necessary for maintaining consistency across downstream analytical systems.

Another major architectural challenge involved balancing flexibility with governance. It is easy to build a script that “works once,” but enterprise healthcare systems require repeatability, auditability, and maintainability. Because of this, the framework evolved to include structured logging, QA validation steps, environment separation strategies, and reproducible export processes. These operational considerations became just as important as the API logic itself.

The project also highlighted the importance of designing systems with future scalability in mind. Even though the original objective focused on RxNorm integration, the architecture needed to remain flexible enough to support future healthcare AI initiatives, predictive analytics projects, and additional interoperability frameworks. Designing with extensibility in mind ensures that engineering work performed today can continue delivering value as organizational needs evolve.

Ultimately, the engineering obstacles encountered throughout this project reinforced an important reality about healthcare technology: building intelligent systems requires far more than simply writing code. It requires architecture planning, operational resilience, scalability strategy, governance thinking, and a deep understanding of how clinical data behaves in real-world environments.

Most importantly, this experience continued reinforcing my broader philosophy surrounding artificial intelligence and healthcare innovation. The purpose of advanced engineering and AI-enabled systems is not to replace healthcare professionals or analysts. The goal is to help them operate more effectively by providing better infrastructure, cleaner data, stronger interoperability, and more intelligent analytical tools. The future of healthcare technology will not be defined solely by automation; it will be defined by how well technology helps people make better decisions.