1995classic paperspaper #10 / 29

The 1995 SQL Reunion: People, Projects, and Politics

by Database luminaries (1995 SQL Reunion)

McJones A reunion of people who worked on System R and its derivatives, including SQL/DS, DB2, and R*, was held at Asilomar on May 29, 1995. This is an edited transcript of the day's discussions, incorporating changes provided by the speakers. It provides an informal but first-hand account of the birth of SQL, the history of System R, and the origins of a number of other relational systems inside and outside IBM. Recommended paired reading: Henry Baker's 1991 letter to the ACM .

Why this paper matters

This paper is the Rosetta Stone for the relational era. It captures the oral history of SQL’s origins through the voices of its creators-Jim Gray, Don Chamberlin, Morton Astrahan, and others-reconstructing the technical and political decisions that shaped IBM’s research from System R to DB2. The 1995 reunion at Asilomar was not just a nostalgic gathering; it was a rare primary source documenting how SQL became the lingua franca of enterprise data, long before ANSI SQL-92 standardization solidified its dominance. In 2026, with SQL still running 98% of global transactional workloads (per DB-Engines), the paper’s transcripts reveal why backward compatibility and declarative semantics won out over navigational models. It exposes the tension between academic rigor (Codd’s relational model) and engineering pragmatism (SQL’s departure from pure relational theory), a conflict that persists today in debates over JSON/relational hybrid systems like CockroachDB and YugabyteDB.

Moreover, the paper’s unfiltered recollections about IBM’s internal politics-competition between System R and Ingres, the rejection of QUEL in favor of SQL, and the rise of DB2-offer a lived case study in how standards emerge from corporate dominance rather than technical merit alone. This raw, human-centered narrative is invaluable for historians and engineers alike, providing context for why PostgreSQL’s optimizer traces its lineage to System R’s query rewrite rules and why even NewSQL systems like CockroachDB emulate DB2’s two-phase commit behavior. Without this paper, the origin myths of SQL would be lost to folklore, replaced by oversimplified narratives that ignore the messy realities of corporate research labs and the engineers who navigated them.

The reunion transcripts also illuminate the early days of distributed databases. IBM’s R* project, discussed in depth in the paper, experimented with distributed query processing and transaction management long before systems like Google’s Spanner or Amazon’s Aurora existed. The paper’s discussion of R*’s challenges-such as handling partial failures, maintaining consistency across sites, and optimizing joins in a distributed setting-mirrors the problems faced by modern distributed SQL engines like Google Spanner and Amazon Aurora. These parallels reveal that the fundamental issues in distributed data management are not new; they are echoes of decisions made in the 1970s and 1980s, repackaged for the cloud era.

Key contributions

Oral history of SQL’s birth: Transcripts from the 1995 reunion document the technical debates (e.g., SEQUEL vs. SQUARE) and IBM’s internal struggles that led to SQL’s adoption over QUEL and other alternatives.
System R’s legacy reconstruction: Firsthand accounts of how System R’s implementation choices-such as the RDS-1 storage engine and the optimizer’s cost model-shaped DB2’s architecture and later open-source systems.
Political and organizational analysis: Explicit discussion of IBM’s internal politics, including the rivalry between San Jose and Yorktown labs, and how corporate decisions overrode technical ones.
Cross-system lineage: Traces the flow of ideas from System R to SQL/DS, DB2, and IBM’s distributed systems like R*, clarifying how distributed query processing emerged from System R’s research.
Standards and compatibility: Insights into how SQL’s early design trade-offs (e.g., allowing duplicates, supporting nulls) enabled rapid adoption but later required painful fixes (e.g., SQL-92’s stricter semantics).
Distributed systems foundations: Highlights IBM’s early experiments with distributed databases (R*), which laid groundwork for modern distributed SQL systems like CockroachDB and YugabyteDB.

Impact on modern systems

The lineage from System R to modern systems is both direct and indirect, visible in query planning, transaction semantics, and even in the DNA of today’s distributed databases. PostgreSQL’s optimizer, for example, still uses a cost-based approach rooted in System R’s dynamic programming algorithm for join ordering. The paper’s transcripts reveal how System R’s developers grappled with index selection and access path optimization-exactly the same problem that modern systems like CockroachDB solve at scale. CockroachDB’s distributed planner inherits not just the SQL grammar but also the System R strategy of decomposing queries into physical operators and estimating costs per node. Similarly, IBM’s own Db2 for z/OS still runs mission-critical financial workloads using an optimizer whose core logic dates back to the System R prototypes discussed in this paper.

The distributed systems angle is even more pronounced in systems like YugabyteDB, which explicitly cites System R’s work in its design docs for distributed SQL. YugabyteDB’s engineers adopted System R’s two-phase commit for its transaction layer, adapting it to a Raft-based consensus layer for fault tolerance. The paper’s discussion of R*’s distributed query processing-where joins across sites were handled via semi-join reducers-echoes in YugabyteDB’s distributed planner today, which minimizes data movement using techniques inspired by System R’s early distributed experiments. Another concrete example is ScyllaDB, a Cassandra-compatible database that reimplements the storage engine using an LSM-tree (see The Log-Structured Merge-Tree (LSM-Tree), 1996) but retains System R’s cost-based optimizer for query planning. ScyllaDB’s choice to separate storage from compute mirrors System R’s early modular design, where the Research Storage System (RSS) handled physical data layout independently of the query processor.

Transaction semantics also trace back to this era. The reunion documents IBM’s decision to support READ COMMITTED isolation by default, a choice that influenced PostgreSQL’s default isolation level and, by extension, the behavior of nearly all open-source databases. Even the concept of MVCC (multiversion concurrency control), now standard in PostgreSQL, ScyllaDB, and TiDB, originated from System R’s research into concurrency control. The paper’s recollection of System R’s lock manager conflicts and deadlock detection algorithms explains why systems like ScyllaDB’s Paxos-based consensus layer still implement lock timeouts and deadlock detection-algorithmic DNA passed down through decades. Microsoft’s SQL Server, for instance, still uses a variant of System R’s ARIES recovery algorithm, which was designed to handle transaction rollbacks and crash recovery efficiently. This persistence of early design choices underscores how foundational systems research can have a decades-long shelf life.

Finally, the paper’s discussion of IBM’s internal politics foreshadows today’s open-core wars. Just as IBM’s Yorktown lab resisted SQL’s adoption due to NIH syndrome, modern vendors like MongoDB and PostgreSQL forks compete over who owns the “true” SQL implementation. The reunion’s transcripts provide a historical lens to understand why PostgreSQL’s SQL conformance outpaces MySQL’s in TPC benchmarks-a direct consequence of IBM’s early investment in SQL’s standardization. Even the rise of Google Bigtable, which eschewed SQL in favor of a simpler API, can be seen as a reaction to the complexity and politics surrounding SQL in the 1990s. Yet, the dominance of SQL in the AI and analytics era proves that the engineering pragmatism of System R’s designers ultimately won out over purist alternatives.

AI era: how LLMs and vector databases relate to this paper

This paper’s insights into declarative query languages and distributed query optimization are suddenly relevant to AI workloads. Vector databases like Pinecone, Weaviate, and Qdrant are essentially SQL’s descendants adapted for embeddings. They inherit the same optimizer challenges: how to route ANN (approximate nearest neighbor) queries across shards, how to estimate selectivity of vector predicates, and how to reduce data movement. The reunion’s transcripts about System R’s optimizer hint at why Pinecone’s HNSW index routing uses a cost model similar to System R’s-they are solving the same problem: “Given a query and a set of indexes, what’s the cheapest way to execute it?” Even pgvector’s optimizer rules, which push down vector similarity functions into storage, trace lineage to System R’s idea of pushing predicates early to reduce I/O.

RAG (Retrieval-Augmented Generation) pipelines depend on semantic indexes-vector databases that store embeddings and support nearest-neighbor queries. These systems are effectively executing SQL-like queries at scale, such as SELECT * WHERE embedding <-> query_vector <-> L2(embedding, query). The reunion’s discussion of IBM’s decision to allow duplicates in SQL tables explains why vector databases like Milvus still allow duplicate vectors and rely on user-defined filters to disambiguate. The AI era has resurrected the same debates about duplicates and nulls that System R’s engineers had in 1974. For example, the CAP Theorem’s trade-offs in distributed vector databases echo System R’s struggles with consistency and availability in R*.

LLM-driven query planning is another frontier. Modern systems like LangChain’s SQL agents use LLMs to rewrite natural language into SQL. The reunion’s transcripts about SQL’s syntax flexibility (e.g., allowing SELECT * in ad-hoc queries) explain why these agents can generate SQL without strict schema enforcement-SQL’s permissiveness enables LLM creativity. But this flexibility comes at a cost: ambiguous queries, poor performance, and security risks (e.g., SQL injection). The paper’s cautionary tales about IBM’s internal disputes over SQL’s syntax mirror today’s debates about prompt injection in LLM-driven query tools. For instance, systems like Redis with its vector search module must balance the need for fast similarity search with the risks of exposing raw vector data to untrusted LLM-generated queries.

Inference latency is impacted by the same optimizer choices discussed in the paper. Systems like Redis’ vector search module and Qdrant use LSM-tree inspired storage (see The Log-Structured Merge-Tree (LSM-Tree), 1996) to support high write throughput for embeddings, but they still rely on a query planner to choose between HNSW, IVF, or brute-force search. The planner’s cost model is directly analogous to System R’s optimizer-estimating the cost of each access path. Even the idea of “yield” from Harvest, Yield, and Scalable Tolerant Systems, 1999 resonates with LLM serving: the system must balance query latency (yield) against embedding update throughput (harvest), a trade-off System R’s engineers faced when tuning System R’s buffer pools. The reunion’s transcripts about System R’s buffer hit ratios are a living lesson for AI vector database designers.

The rise of embeddings serving as a distinct workload category also highlights the paper’s relevance. Systems like Vespa and Zilliz must handle two types of queries: traditional CRUD operations and vector similarity searches. Their planners must decide when to route a query to a traditional SQL engine versus a vector index, a decision that mirrors System R’s early work on access path selection. The eventually consistent nature of many vector databases further underscores the paper’s themes, as these systems prioritize availability and partition tolerance (per the CAP Theorem) over strong consistency-a direct echo of System R’s debates about when to enforce serializability versus allowing relaxed isolation levels.

The 1995 SQL Reunion: People, Projects, and Politics

Why this paper matters

Key contributions

Impact on modern systems

AI era: how LLMs and vector databases relate to this paper

Further reading