Farhan Ali Khan

Database startup: a multi-model database (document, graph, vector). Led the storage effort.

Storage engines and team

• Led the storage effort, working on optimisations and on what the company needed from storage.
• Built SurrealKV, an LSM-tree key-value store in Rust, from scratch, and made it performant.
• Worked on a copy-on-write B+ tree key-value store, built internally for our use case.
• Worked on performance improvements in RocksDB and at the storage-engine level.
• Did low-level optimisation on in-memory Rust crates such as vart (a versioned adaptive radix trie) and several others.

Distributed and cloud storage

• Added object storage support on top of our LSM engine, to build a lakebase architecture for the company's growing needs.
• Worked on our distributed transactional store, built on the TAPIR paper (Zhang et al., SOSP 2015), and did deterministic simulation testing (DST) across our internal products.

More on TAPIR

TAPIR, the Transaction Application Protocol for Inconsistent Replication (Zhang et al., SOSP 2015), builds linearizable distributed transactions on replication that gives no consistency guarantees of its own. Consistency is enforced only at the transaction layer, which removes coordination from the replication layer.

AI and agent memory

• Worked on SurrealDB's agent memory product.
• Worked on understanding facts and entities and the logic around them, and on defining ontologies for the product.
• Used an agentic workflow with TLA+ modelling before writing code, then Kani checks, Verus proofs, and property and mutation testing.

More on agent memory and ontologies

The A-MEM paper (Xu et al., NeurIPS 2025) is related research on agent memory. It stores each memory as a note with its content, keywords, tags, a short context and an embedding, links related notes, and lets a new memory update older notes.

An ontology says which kinds of entities exist, which relations can hold between them, and which rules a fact has to follow. Defining one for the product gives facts and entities a structure to map onto, which keeps memory consistent as more is added.

Startup. immudb: open-source immutable database with cryptographic (Merkle tree) verification; tamper-resistant and auditable.

• Worked on data retention, implementing efficient deletes in the database, where proofs and storage are interlinked.
• Worked on compaction and on improving key-value store performance.
• Developed new features, maintained core components and tested the database.
• Engaged with the immudb community.

Large company. Internal developer platform running many Kubernetes clusters with thousands of nodes under PCI, HIPAA and FedRAMP High compliance, with separate clusters for development, staging, production and sensitive workloads across data centres.

• Built Kubernetes operators for the platform, including an operator for multi-cluster management across clouds that gave tenants Kubernetes federation and multi-tenancy.
• Designed and built a self-service model for platform-managed services: tenants deploy curated Kubernetes apps (Kafka, Vault, the Prometheus monitoring stack) with the team's deployment practices built in, instead of installing and running each one from scratch.
• Managed platform releases through a Kubernetes-native release operator built by the team.
• Took SRE on-call shifts, which showed where tenants struggled and kept me hands-on with their issues.
• Introduced an end-to-end (E2E) testing framework and behavioural tests written from the tenant's point of view, so operators were tested before tenants used them.
• Improved the team's peer code review process to raise code quality and collaboration.

Large company. Containerised firewall service that automates network operations and monitors customer networks, with fast router onboarding and near real-time configuration updates.

• Worked on moving firewall deployments from a centralised architecture to a decentralised peer-to-peer one, with a Go management layer running alongside the Snort firewall on each router.
• Researched group membership algorithms and built a gossip-based membership mesh across routers on SWIM (Scalable Weakly-consistent Infection-style Process Group Membership), using HashiCorp memberlist.
• Built an in-memory shared key-value store for configuration and version updates, gossiped across the cluster and ordered with Lamport clocks.
• Designed a more robust successor that models the data as CRDTs (conflict-free replicated data types) to remove clock-skew problems.
• Built core STS services: designed the gRPC APIs, coordinated with other teams, planned deployment strategies for testing the distributed setup, and guided the team on concurrent Go, design patterns and testing strategy.

Ad-tech startup: an ad platform for mobile games serving more than 500 million ad requests a day. Led the data engineering effort and worked closely with business and operations teams.

Ad serving

• Designed the real-time bidding (RTB) protocol: each request broadcasts bid requests to bidders (AdMob, Kraken, Facebook and others), evaluates responses, notifies the winner and serves the ad markup.
• Researched RTB papers and designed an asynchronous flow that prepares ads for subsequent requests, keeping every response under 500 ms.
• Set up the microservice infrastructure on Google Cloud (Kubernetes, Terraform, Consul) with a caching layer on Redis and Aerospike.
• Reduced infrastructure spend through algorithm and caching optimisations while holding the latency target.
• Built the monitoring stack (Prometheus, Grafana) with real-time cache dashboards the business used for scaling decisions. Slides: Ad serving platform using Redis (RedisConf 2018).

Data engineering (lead)

• Led the data engineering effort: built the company's first data pipelines on Kafka and BigQuery, ingesting more than 2 TB a day.
• Converted raw gzipped data to Parquet and Cap'n Proto to make transfers cost-effective, and optimised BigQuery queries with the data team.
• Cached query results in Redis data structures for faster queries; optimised Aerospike queries and cost, giving the operations team data to choose which campaigns to run during the day.
• Built BigQuery dashboards that surfaced errors in the Android SDK.
• Set up a highly available MySQL cluster on GCP with replication and backups for critical data, and a data lake on Amazon Glacier with pipelines for querying historical data.

Fintech startup: a corporate expense management product. Led infrastructure and worked directly with customers.

• Worked directly with end customers.
• Designed and built role-based access control (RBAC), a core feature that manages the hierarchy of users in each customer organisation.
• Built and managed integrations with third-party APIs, and the public API HapPay exposed to partners (Dunzo integrated with it in 2016).
• Built accounting integrations that sync customers' expenses into SAP, Oracle, QuickBooks, Tally and other ERP tools.
• Built an engagement workflow engine (DAG / state machine) to track the customer lifecycle, scaling the in-house streaming pipeline, Elasticsearch and scheduler.
• Led infrastructure for the product: Docker and AWS, log management and analysis across services on the ELK stack, and scaling, monitoring and migrating the critical MySQL and PostgreSQL databases.

• Built custom SharePoint 2013 and 2010 solutions in Visual Studio for enterprise clients: web parts, site definitions, designer workflows, event receivers and timer jobs, with HTML5, CSS3 and JavaScript.
• Customised client applications with the server and client object models, delivered enhancements and change requests, and debugged critical defects.