Exploring Semi-supervised Variational Autoencoders
for Biomedical Relation Extraction
Yijia Zhanga,b and Zhiyong Lua*
a National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes
of Health (NIH), Bethesda, Maryland 20894, USA
b School of Computer Science and Technology, Dalian University of Technology, Dalian, Liaoning 116023, China
Corresponding author: Zhiyong Lu (zhiyong.lu@nih.gov)
Abstract
The biomedical literature provides a rich source of knowledge such as protein-protein interactions (PPIs), drug-drug
interactions (DDIs) and chemical-protein interactions (CPIs). Biomedical relation extraction aims to automatically
extract biomedical relations from biomedical text for various biomedical research. State-of-the-art methods for
biomedical relation extraction are primarily based on supervised machine learning and therefore depend on (sufficient)
labeled data. However, creating large sets of training data is prohibitively expensive and labor-intensive, especially
so in biomedicine as domain knowledge is required. In contrast, there is a large amount of unlabeled biomedical text
available in PubMed. Hence, computational methods capable of employing unlabeled data to reduce the burden of
manual annotation are of particular interest in biomedical relation extraction. We present a novel semi-supervised
approach based on variational autoencoder (VAE) for biomedical relation extraction. Our model consists of the
following three parts, a classifier, an encoder and a decoder. The classifier is implemented using multi-layer
convolutional neural networks (CNNs), and the encoder and decoder are implemented using both bidirectional long
short-term memory networks (Bi-LSTMs) and CNNs, respectively. The semi-supervised mechanism allows our model
to learn features from both the labeled and unlabeled data. We evaluate our method on multiple public PPI, DDI and
CPI corpora. Experimental results show that our method effectively exploits the unlabeled data to improve the
performance and reduce the dependence on labeled data. To our best knowledge, this is the first semi-supervised VAE–
based method for (biomedical) relation extraction. Our results suggest that exploiting such unlabeled data can be
greatly beneficial to improved performance in various biomedical relation extraction, especially when only limited
labeled data (e.g. 2000 samples or less) is available in such tasks.
Keywords: Biomedical Literature; Relation extraction; Semi-supervised learning; Variational autoencoder
1. Introduction
Currently there are over 28 million articles in PubMed, and each year the biomedical literature grows by more than
one million articles [1, 2]. As a result, a vast amount of valuable knowledge about proteins, drugs, diseases and
chemicals, critical for various biomedical research studies, is “locked” in the unstructured free text [3, 4]. With the
rapid growth, it is increasingly challenging to manually curate information from biomedical literature, such as protein-
protein interactions (PPIs), drug-drug interactions (DDIs) and chemical-protein interactions (CPIs). The goal of
information extraction in biomedicine is to automatically extract biomedical relations through advanced natural
language processing (NLP) and machine learning techniques.
Over the past decade, a number of hand-annotated datasets have been created for biomedical relation extraction
such as the various PPI corpora [5] and DDI 2013 corpus [6]. Based on these public corpora, a number of methods [7-
10] have been attempted. For example, Airola, et al. (2008) proposed an all path kernel approach to extract PPIs based
on the lexical and syntactic features from the dependency syntactic graph. Zhang, et al. (2012) proposed a hash
subgraph pairwise kernel method to extract DDIs, which efficiently generates hash features from a dependency
syntactic graph based on the hash operation. More recently, models based on deep neural networks such as
convolutional neural networks (CNNs) and recurrent neural networks (RNNs), have shown promising results in
various tasks [11–13].
Most of the aforementioned high-performing systems in biomedical relation extraction to date are based on
supervised machine-learning approaches, which are known to be dependent on manually labeled data. Although data
exist for few relation types such as PPIs and DDIs, there is a lack of large–scale training data for many other critical
relationships in the biomedical domain, including gene-disease, drug-disease, and drug-mutation, due to the
prohibitive expense of manual annotation. In response, alternative methods such as distant supervision have been
proposed [14, 15], which automatically create training data based on existing curated biological databases. However,
since the databases are generally incomplete and our language is highly rich and diverse, such automatically created
labeled data are always noisy [16].
Fig. 1. An illustrative example of a semi-supervised learning method. The white, black and gray circles represent
positive, negative and unlabeled relation instances, respectively. The dashed line represents the decision boundary
decided by the classifier.
In this paper, we introduce a novel semi-supervised variational autoencoder (VAE)-based method for biomedical
relation extraction. In contrast to supervised learning, the semi-supervised method learns discriminative features from
both labeled and unlabeled data. Fig. 1 represents an illustrative example of the difference between supervised and
semi-supervised learning methods. It is difficult to predict the decision boundary accurately based on a small number
of labeled instances. However, if we integrate the unlabeled instances – generally distributed according to a mixture
of individual-class distributions – with the labeled ones, we may considerably improve the learning accuracy. Given
the great abundance of freely available texts in PubMed, it is particularly valuable to explore semi-supervised-based
methods for improving biomedical relation extraction.
Motivated by the recent success of semi–supervised VAE methods in image classification [17, 18], text modeling
[19] and text classification tasks [20], in this work we investigate its feasibility for relation extraction, which differs
significantly from the other tasks. To the best of our knowledge, this is the first VAE-based method for (biomedical)
relation extraction. To demonstrate its robustness, we validate our method on multiple different biomedical relation
types: PPIs, DDIs and CPI extraction.
2. Materials and methods
2.1 Biomedical Relation Extraction
Biomedical relation extraction is generally approached as the task of classifying whether a specified semantic relation
holds between two biomedical entities within a sentence or document. According to the number of semantic relation
classes, biomedical relation extraction can be further categorized into binary vs. multi-class relation extraction.
In this paper, we focus on PPI, DDI and CPI extraction. PPI is a binary relation extraction task, whereas DDI and
CPI are multi-class relation extraction task. We show some examples as follows.
PPI extraction example: These results suggest that profilin may be involved in the pathogenesis of
glomerulonephritis by reorganizing the actin cytoskeleton.
DDI extraction example: The concomitant administration of gemfibrozil with Targretin capsules is not
recommended.
CPI extraction example: Compound C diminished AMPK phosphorylation and enzymatic activity, resulting in
the reduced phosphorylation of its target acetyl CoA carboxylase.
In the case of PPIs, a system only needs to identify whether the candidate entity pair has a semantic relation or not.
For DDI and CPI, a system requires not only the detection of the semantic relation between two candidate entities but
also the classification of the specific semantic relation into the correct type. For example, the DDI extraction task
requires a system to distinguish five different DDI types, including Advice, Effect, Mechanism, Int and Negative.
Similarly, the CPI extraction task includes six specific types: Activator, Inhibitor, Agonist, Antagonist, Substrate and
Negative.
2.2 Datasets
For PPI, we used the BioInfer dataset [21]. For DDI, the DDI extraction 2013 corpus [6, 22] was used. For CPI, we
use the recent ChemProt corpus [23], which was served as the benchmarking data in the BioCreative VI challenge
task. The detailed statistics of the datasets are listed in Tables 1, 2 and 3, respectively.
Table 1. The statistics of the PPI corpus
Dataset
Sentences
Positive
Negative
Total
BioInfer
1100
2,534
7,132
9,666
Table 2. The statistics of the DDI corpus
Dataset
Advice
Effect
Mechanism
Int
Negative
Total
Training set
826
1,687
1,319
188
23,772
27,792
Test set
221
360
302
96
4,737
5,716
Table 3. The statistics of the ChemProt corpus
Relations
Training set
Development set
Test set
Active
768
550
664
Inhibitor
Agonist
170
116
194
Antagonist
234
197
281
Substrate
705
457
643
Negative
Total
