Blog 3: Message Passing Meets Meaning: Graph Neural Networks for KGs

So far, we’ve seen how knowledge graphs (KGs) capture relationships through symbolic triples, and how embeddings translate those into vector space. But now, we enter a new level of expressiveness, when each node learns not just from its own embedding, but from its neighbors too.

That’s the heart of Graph Neural Networks (GNNs).

Why GNNs on Knowledge Graphs?

Imagine a node representing Albert Einstein. It connects to:

• born_in → Ulm
• won → Nobel Prize
• field → Physics

Shouldn’t our model consider these neighbors to understand what Einstein means in the graph? That’s exactly what GNNs do, through message passing.

Message Passing Intuition

Each node updates its vector by aggregating information from its neighbors. Formally, at each layer \( l \), we compute:

\[ \mathbf{h}_v^{(l+1)} = \text{Update}\left(\mathbf{h}_v^{(l)}, \text{Aggregate}\left( \{ \mathbf{h}_u^{(l)} : u \in \mathcal{N}(v) \} \right) \right) \]

where \( \mathcal{N}(v) \) is the set of neighbors of node \( v \).

This allows information to flow across the graph, gradually incorporating broader context.

Types of GNNs Used in KGs

While classic GNNs (like GCN, GraphSAGE) treat all edges equally, KGs have typed, directional edges. So we need Relational GNNs.

1. R-GCN (Relational Graph Convolutional Network)

Extends GCN by assigning separate weights per relation type:

\[ \mathbf{h}_v^{(l+1)} = \sigma\left( \sum_{r \in \mathcal{R}} \sum_{u \in \mathcal{N}_r(v)} \frac{1}{c_{v,r}} \mathbf{W}_r^{(l)} \mathbf{h}_u^{(l)} + \mathbf{W}_0^{(l)} \mathbf{h}_v^{(l)} \right) \]

Here:

• \( \mathcal{R} \) = set of relations
• \( \mathcal{N}_r(v) \) = neighbors via relation \( r \)
• \( \mathbf{W}_r \) = relation-specific weight matrix

Output: contextualized node embeddings.

2. CompGCN (Composition-based GCN)

Goes a step further, updates both entity and relation embeddings:

\[ \mathbf{h}_v^{(l+1)} = \sigma\left( \sum_{(u, r) \in \mathcal{N}(v)} \mathbf{W}_r^{(l)} \phi(\mathbf{h}_u^{(l)}, \mathbf{r}) \right) \]

where \( \phi \) is a composition function (e.g., addition, multiplication, circular correlation).

This allows the model to model relation-specific transformations in message passing.

Graph-Level Tasks

Once we’ve trained a GNN on a knowledge graph, we can apply it to tasks like:

• Link prediction: predict missing triples based on context-aware embeddings
• Entity classification: infer types or categories for entities
• Relation extraction: from textual graphs, derive missing links

Layer Stacking and Over-Smoothing

Unlike standard MLPs, GNNs cannot stack unlimited layers. Why?

Because node embeddings become too similar, an effect called over-smoothing. To counter this:

• Use skip connections
• Restrict to 2–3 layers
• Apply normalization or attention

Table: Relational GNN Comparison

\[ \begin{array}{|c|c|c|c|} \hline \textbf{Model} & \textbf{Learns Relations?} & \textbf{Edge-Aware?} & \textbf{Strength} \\ \hline \text{GCN} & \text{No} & \text{No} & \text{Simple, fast} \\ \hline \text{R-GCN} & \text{Yes (fixed)} & \text{Yes} & \text{Captures relation structure} \\ \hline \text{CompGCN} & \text{Yes (dynamic)} & \text{Yes} & \text{Entity + Relation co-updates} \\ \hline \end{array} \]

TL;DR

• GNNs allow entities to learn from their neighbors via message passing.
• Relational GNNs respect typed edges in KGs.
• They power smarter embeddings, enabling structure-aware reasoning.

Coming Up Next:

Blog 4: "Reasoning Over KGs: From Rule Mining to Neural Deduction"

We’ll go beyond embeddings and dive into actual reasoning, how neural-symbolic models perform inference, generalize logical rules, and combine the best of logic + deep learning.