Ask a seasoned scientist how to interpret a biological result, and the answer usually starts with “it depends.” What comes after is the valuable part: a set of nuances and judgment calls built up over years of real life experience.
The idea that data is the moat has been the sentiment for decades, but in the AI era it takes on a new meaning. The moat goes beyond files. It’s the tacit knowledge that guides how you reason over your data. This unspoken knowledge is traditionally fragmented, across slide decks, documents and in people’s heads.
Fragmentation is painful but tolerable when gaps can be filled by scientists with lived experience. When the same framework is handed to an agent and the gaps get filled by a model’s biases instead of your team’s, silently, on every run.
This leaves two questions worth answering before you put agents anywhere near a portfolio decision:
What does a portfolio look like after two years of decisions made on reasoning no one can trace?
Where should your institutional knowledge live so that doesn’t happen?
Institutional memory for agentic systems
There are only three places your reasoning can live. In someone’s head, where it can’t be executed. In a document or deck, where it gets reinterpreted every time it’s read. Or in infrastructure an agent can run and you can audit afterwards.
The third option is institutional memory for agents. It is infrastructure that lets you encode, record and govern your tacit knowledge while staying model and agent agnostic. It connects to your multimodal data and turns your historical learnings into tools you can apply to future decisions.
Keeping it separate from the model solves three problems:
Model flexibility and vendor lock.
Every day a new model or agent is announced, and policy and sentiment around the providers are evolving just as fast. As teams test, iterate and select which providers they would like to work with, they need the flexibility to take their institutional knowledge with them.
Getting your agents to think and reason like you.
When using an AI agent for a task that you have deep domain expertise in, the gap between your tacit knowledge and an AI agent’s pre-training bias becomes painfully obvious.
Think about the number of times you have asked a question, looked at the answer and disagreed because of the nuance the agent failed to take into consideration. If agentic systems are to be trusted with critical decisions in an industry with no room for error, they must think and reason like the best scientists in the organization.
Protecting your most valuable asset.
The institutional knowledge that drives your decision making is one of the most valuable assets a company owns. This knowledge should be intentionally documented, preserved and used to drive decisions.
Why skills files aren’t enough
I know what you’re thinking: armed with an arsenal of skills files, all of these problems disappear.
The challenge with solely relying on skills files is the solution does not scale when teams are evolving alongside complex data and decisions. It’s a short term solution that degrades over time.
Here’s where it crumbles in practice
Someone writes down how the best scientists evaluate a target, saves it as a skill file, and hands it to the model at the start of every task. It’s the closest thing to institutional memory you can stand up in an afternoon, and for a while it works.
One file becomes ten and additional portfolio evaluation decisions beyond target prioritization need to be made. Frameworks change, and now there are three versions of the same file and no one is sure which one is canonical.
The reasoning is written in plain English, so the model interprets it a little differently on each run. A framework written in prose is full of soft instructions. “Give a higher weight to genetic evidence” needs to be transformed into a computation. “When the data is contradictory” is an important judgement call the model will make on its own.
When it’s time to revisit a call made five months ago, identifying which decision frameworks were used, what the prompt was, and where the underlying data is turns into a forensic investigation.
These are maintenance complaints that cause real bottlenecks. A bottleneck slows a drug program down, but a bad decision cripples it. As teams put AI in the decision loop, what matters is whether the system reasons like their team, and how it holds up as the data and the decisions accumulate.
Teams need a robust and scalable way to capture their decision making logic. This logic needs to be data and schema aware. This system must be capable of connecting to any agentic system. This is exactly why we built the BioBox MCP.
BioBox MCP: Agentic Institutional memory
The BioBox platform is infrastructure for scientific reasoning. It enables organizations to operationalize their tacit knowledge and use it to drive decisions.
A multimodal data foundation. Integrated multi modal data used to inform decisions. Scientists, human and AI, need an easy way to access all of their data in one connected ecosystem. The backbone of the data foundation is a custom knowledge graph purpose built for each client, composed of proprietary and public data.
Reasoning System and decision modules. The logic and tradeoffs used to make historical decisions and drive future ones. Think of it as the git for your most complex decisions, ready to be leveraged by your human scientists or AI co-scientists of choice. These decision frameworks connect directly to the data foundation and graph schema.
Decision-ready interfaces and integrations. An ecosystem for cross-functional decision making. This is where the reasoning and data meet the actual decision. A scientist reviews a call and sees the exact logic that produced it. R&D teams can make portfolio decisions with confidence by leveraging historical learnings backed by data.
The MCP connects agentic systems to the BioBox infrastructure. The BioBox MCP was put to the test against skills files to find out if an agentic institutional memory truly makes a difference when it comes to AI in the loop decision making.
The Test: BioBox MCP vs Skills Files
The same portfolio evaluation decisions were executed across two setups
Comprehensive skills files were curated using k-dense-ai/claude-scientific-skills. Skills Files were enriched with Cypher queries to retrieve the data from the graph with a higher degree of accuracy than a standard plain English skills file.
Portfolio Decisions
Target prioritization: Prioritize targets for Psoriasis according to a predefined specific scientific framework.
Indication prioritization: Prioritize indications according to a predefined commercial and biological strategy
Multi-modal questions: Answer questions that span multiple data modalities.
Gap analysis: Surface why decisions have changed over time.
The knowledge graph. The graph was composed of 52.1M objects and 839M edges. This graph integrated a variety of multi-omic data including: single-cell differential expression data, Open Targets genetics, ClinVar genetic associations, ChEMBL drug properties, NIH clinical trials, GTEx normal tissue expression, DepMap CRISPR screens, GO biological processes, Reactome pathways, disease ontologies and global disease burden data. Every client on the BioBox platform curates their own graph; this one was built for the test.
The link to the full whitepaper is at the bottom of the article.
The results
Both setups ran a full session of 30+ questions against the same graph. A shared set of identical questions was scored side by side.
Verbatim transcripts from both sessions were scored by Opus 5, using Anthropic’s Skills Grader Agent. This eval forces a pass or fail on each test with cited evidence. Every question was re-evaluated directly against the graph.
The sessions produced 19 pass/fail tests because one answer can fail in several ways. Three of the criteria evaluated what changed between two versions of a scientific framework and how this impacted the downstream decisions.
Long term
Two things don’t show up in a single session and matter more than anything above.
Scalability. Can this be maintained by cross functional teams as data and knowledge grow?
Governance and auditability. How easy it for teams to look back on previous decisions to understand how and why they were made?
What this looked like in practice
One example per decision type.
Target Prioritization
Reproducibility under a moving substrate.
The task: Rank and prioritize psoriasis targets using strictly defined multi-modal criteria in a skills file or a BioBox Reasoning System accessible via MCP.
Mid-session, a single differential expression edge was removed from the graph. It moved a target from rank 3 to rank 244. The skills file setup failed to detect this.
When asked to rerun the target prioritization rankings, it returned the results stored in cache rather than live querying the graph.
The failure was silent and caught by luck. Despite being reminded to query the graph, the agent confidently asserted that there was no need to and the graph was read only.
It was only discovered because an unrelated question about upregulated genes was asked and the numbers didn’t add up. The significant-DE gene list had been reduced from 4,872 to 4,871. Exactly one gene was removed, and it was the one previously sitting at rank #3.
Ingesting new data and revising decision criteria is a standard step in hypothesis testing, not an edge case.
The MCP session ran under the same challenging conditions. The graph and reasoning frameworks were actively updated in real time. It reported the deprioritized targets without being prompted and pinned every answer to a specific report snapshot.
Indication prioritization
Six of the top ten skills file “indications” weren’t actionable
The task: Prioritize indications using two different frameworks and specify where AML ranks.
The skills setup took 12m 40s and was the slowest answer in the session. It had to build its own disease universe and landed on 6,639 diseases. Its top 10 included neoplasm, disease, genetic disorder, carcinoma, autosomal recessive disease and hereditary neoplastic syndrome.
Six of the ten are ontology parent nodes. They rank because they inherit evidence counts from everything beneath them. You cannot run a program against “disease.”
The MCP took 4m 09s and scored 12,533 diseases, nearly twice the universe because that universe is defined in the framework rather than reconstructed at runtime. Every entry in its top 10 was something a portfolio committee could actually discuss.
Deciding what’s in the running is scientific judgment. It should be defined by the framework, not by whichever agent happens to be answering.
Multi-modal data interrogation
Ontology overrides resulted in downstream consequences.
The task: Answer a set of biological questions using multi-modal datasets.
The agent was asked which genes are upregulated in psoriasis macrophages and also carry a genetic risk variant for psoriasis. The skills setup returned zero genes and explained the zero as a biology artifact
“This is a real biological pattern, not a query artifact… The graph keeps these two axes cleanly separate for macrophages.”
It supported this assertion with a table explaining that the genes with a psoriasis variant are signalling and regulatory, while macrophage upregulated genes are effector and antimicrobial. Despite sounding convincing, this was not correct.
The correct answer in the example graph was 172 genes, including TYK2, IL2RA, ERAP1, TNFRSF1A, CARD9, IL17RA, STAT3, TLR4, ETS1, RUNX1.
The entire overlap was below the threshold the agent had chosen rather than what was defined in the ontology. It then explained the absence of a result as a property of psoriasis biology.
The MCP returned the correct answer twice, four days apart.
Gap analysis & Traceability
Four gap analysis questions, zero answers from the agent equipped with skills.
The task: The agent was asked what prioritization criteria had changed and which targets moved as a result.
The MCP delta report tool provides the agent with access to live and archived report comparisons. The agent listed the three criteria that contributed to the deprioritization; macrophage expression was removed entirely, suprabasal keratinocyte expression and interferon signalling were given higher importance.
Then it listed the targets that had moved and tied each to a specific change. TRAF1 fell 6,841 ranks, and GLDN fell 2,444. It also outlined the updated graph queries and scoring formulas responsible for both. That separated “someone added new prioritization criteria” from “someone changed how the score is calculated”. This is a distinction required for program review, and one that can’t be reconstructed after the fact from a chat log.
The skills setup couldn’t answer this.
A skills run produces a CSV in a working directory and a chat log. There is no version, snapshot, or record of what the framework looked like last quarter. Its version control is the filesystem and its memory is the context window.
The MCP Gap analysis can compare multiple decision snapshots over time. The skills setup only ever holds one.
Critiques & Limitations
A longer horizon evaluation is needed. We are currently working on a second evaluation with a longer time horizon and 2+ sessions per setup.
Some failures belong to the harness. Defaulting to cached data is typical agent behaviour. However, the agent is far less likely to rely on cached data when using the MCP, as it always requires a live query of the graph.
The BioBox team built both setups. The BioBox team curated the graph, MCP, reasoning systems, and the skills files used in this evaluation. Skills files were robust and backed by ontology aware Cypher. Responses from both setups were evaluated against the graph using Opus 5.
Configurations, questions, transcripts, rubric and adjudication queries are available in our white paper.
Areas of improvement for the MCP
The agent treats the MCP tools as the ultimate source of truth. Meaning that if there is a user error in the reasoning system, the agent is likely to execute the human error as described without question. To improve the MCP, additional validation checks have been added to ensure the intended decision frameworks provided by the user are accurately reflected.
Throughout this evaluation, the agent was more likely to inject its pre-training biases when equipped with skills files. Consequently, this increases the odds of the agent overriding predefined frameworks. However, this can result in the agent catching errors in the skills files provided.
What I’d take from this
A skills file encodes a description of your framework. A reasoning system is the framework. To use a skills file, an agent has to translate your description into something executable and construct a candidate universe before anything is executed. The failure isn’t in the file; it’s that a description has to be translated before it runs, and the translator is the model.
Those steps are judgment calls, and across the session, they were made differently. Each call was defensible and appeared to make sense on first glance. The problem is they did not accurately reflect the pre-defined scientific criteria consistently over time. If you go the extra mile to write Cypher into your skills files, they are one graph schema change away from being obsolete.
Skills files have their time and place. Great for simple one off analyses but fall short when it comes to complex cross-functional long term decision making.
Reasoning has to be a first class artifact. The moment a decision needs to be revisited, defended or compared, you need a versioned snapshot. A chat log does not capture this. Managing, versioning and updating multiple skills files and the artifacts of each analysis without an MCP was a nightmare.
Your institutional knowledge should sit on infrastructure you own. Judgement calls and decision frameworks are what separate a good company from a great. Everyone asks whether they can trust the model with their data. No one asks whether they can trust it with their reasoning.
A skills file lets an agent approximate your scientific framework. The BioBox MCP lets it execute your scientific frameworks on your custom data foundation. It lets you track which version it ran, backed by the data and knowledge available at the time of the decision.
When it comes to complex biological decisions, the final answer is usually still “it depends.” The point is being able to consistently execute and retrieve what it depended on.
Download our whitepaper here
Sources
claude-scientific-skills — k-dense-ai
Skills Grader Agent — Anthropic