Entity Matching with Similarity Maps and Deep Learning

The example concisely introduces the basic concepts and functionality of the package. Its goals are (i) to bring the reader up to speed with how to use the package to fit entity-matching models and (ii) to familiarize the reader with the basic concepts that are used in the documentation and naming conventions of the package. For simplicity, we use a deep learning matching model for this example. Examples of neural-symbolic matching models are given in the Neural-symbolic Entity Matching and Reasoning vignettes.

Prerequisites

Load the libraries we will use and set the seed for reproducibility.

from neer_match.matching_model import DLMatchingModel
from neer_match.similarity_map import SimilarityMap
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import random
import tensorflow as tf

random.seed(42)
np.random.seed(42)
tf.random.set_seed(42)

Preprocessing

The preprocessing stage aims to bring the data in a form compatible with the neermatch package’s calling conventions. The package expects the data to be in the form of two data frames, left and right, that contain the entities to be matched. The package also expects a third data frame, matches, that contains the indices of the matching entities in the left and right data frames. We illustrate the calling convention by constructing a toy example.

Left Data Set

The data used in this example is constructed from a subset of the game_reviews dataset shipped with the R version of the package (see the game_review documentation). The subset is created by selecting all the records in game_reviews having titles starting with either "Metal Slug" or "Metal Gear". The selection results are hard-coded here to keep the example code self-contained. Following the package’s naming convention, we name the resulting data frame left.

left = pd.DataFrame(
    {
        "title": [
            # fmt: off
            "Metal Gear Solid 4: Guns of the Patriots", "Metal Gear Solid",
             "Metal Gear Solid 3: Snake Eater", "Metal Gear Solid V: The Phantom Pain",
             "Metal Gear Solid Mobile", "Metal Gear Solid: Portable Ops",
             "Metal Gear Solid 2: Substance", "Metal Gear Solid: The Twin Snakes",
             "Metal Gear Solid", "Metal Gear Rising: Revengeance",
             "Metal Gear Solid Integral", "Metal Gear Solid: Peace Walker HD Edition",
             "Metal Slug 4 Mobile", "Metal Gear Rising: Revengeance",
             "Metal Gear Online", "Metal Gear Acid",
             "Metal Slug Anthology", "Metal Gear Solid V: Ground Zeroes",
             "Metal Slug Mobile", "Metal Gear Rising: Revengeance - Jetstream",
             "Metal Slug 2", "Metal Gear Solid V: Ground Zeroes",
             "Metal Slug 7", "Metal Gear Solid: VR Missions",
             "Metal Slug 4 & 5", "Metal Gear Rising: Revengeance - Blade Wolf",
             "Metal Gear Acid 2", "Metal Slug XX",
             "Metal Slug Defense", "Metal Slug 1",
             "Metal Gear Solid V: Metal Gear Online", "Metal Slug Touch",
             "Metal Slug X", "Metal Gear Rising: Revengeance - Jetstream",
             "Metal Gear Survive", "Metal Gear",
             # fmt: on
        ],
        "platform": [
            # fmt: off
            "PS3", "PS", "PS2", "XONE", "MOBI", "PSP", "XBOX", "GC", "PC", "PC", "PS", "PS3",
             "MOBI", "X360", "PS3", "PSP", "PS4", "PS4", "MOBI", "PS3", "IOS", "XONE", "DS", "PS",
             "XBOX", "X360", "MOBI", "PSP", "IOS", "IOS", "PS4", "IOS", "IOS", "X360", "PS4", "MOBI",
             # fmt: on
        ],
        "year": [
            # fmt: off
            2008, 1998, 2004, 2015, 2008, 2006, 2002, 2004, 2000, 2014, 1999, 2012, 2008, 2013,
            2008, 2005, 2016, 2014, 2004, 2013, 2013, 2014, 2008, 1999, 2005, 2013, 2009, 2010,
            2014, 2012, 2015, 2009, 2013, 2013, 2018, 2004
            # fmt: on
        ],
        "scores": [
            # fmt: off
            93.53, 93.24, 91.77, 90.38, 92.50, 86.95, 86.66, 85.58, 84.22, 83.55, 90.00, 90.00,
            90.00, 82.56, 80.37, 76.70, 80.00, 75.23, 75.00, 75.00, 77.00, 72.50, 72.11, 70.64,
            70.47, 70.00, 70.00, 68.97, 70.00, 68.33, 67.50, 60.00, 64.00, 60.00, 61.09, 50.00
            # fmt: on
        ],
        "reviews": [
            # fmt: off
            85, 29, 86, 12, 2, 60, 56, 69, 18, 11, 1, 1, 1, 31, 8, 52, 1, 48, 4, 2, 5, 6, 42, 18,
            33, 2, 1, 18, 6, 3, 2, 1, 5, 1, 33, 1
            # fmt: on
        ],
        "developer": [
            # fmt: off
            "Kojima Productions/Konami", "KCEJ/Konami", "KCEJ/Konami", 
            "Kojima Productions/Konami", "Ideaworks3D/Konami", "Kojima Productions/Konami", 
            "KCEJ/Konami", "Silicon Knights/Konami", "Digital Dialect/Konami",
            "PlatinumGames/Konami", "Konami", "Kojima Productions/Konami", "SNK Playmore/I-Play",
            "PlatinumGames/Konami", "Kojima Productions/Konami", "Konami",
            "Terminal Reality/SNK Playmore", "Kojima Productions/Konami", "SNK Playmore/I-Play",
            "PlatinumGames/Konami", "SNK Playmore", "Kojima Productions/Konami",
            "SNK Playmore/Ignition Entertainment", "KCEJ/Konami", "BrezzaSoft/SNK Playmore",
            "PlatinumGames/Konami", "Konami/Glu Mobile", "SNK Playmore", "SNK Playmore",
            "SNK Playmore", "Kojima Productions/Konami", "SNK Playmore", "SNK Playmore",
            "PlatinumGames/Konami", "Konami", "Konami Mobile & Online, Inc./Konami"
            # fmt: on
        ],
    }
)

Right Data Set

What is the right data frame in this example? We construct the right data frame by copying the left data frame and introducing noise in the title and developer columns. Up to three characters are randomly removed from the title column, and up to two characters are randomly removed from the developer column.

In addition, we create three duplicate matching records on the right data frame to illustrate that the matching models of the package can be used to link datasets with records/entities having either one-to-many or many-to-many relations.

By construction, the left and right data frames in our examples have the same columns. In practice, this is not always the case. For such cases, the neermatch package supports specifying different columns of the left and right data frames. To illustrate how the package can be used in such cases, we rename the developer column in the right data frame to dev.

right = left.copy()
no_duplicates = 3
right = pd.concat([right, right.sample(no_duplicates)])

tpos = right.columns.get_loc("title")
dpos = right.columns.get_loc("developer")

for r in range(right.shape[0]):
    no_chars = random.randint(1, 3)
    title = right.iloc[r, tpos]
    right.iloc[r, tpos] = "".join(
        [
            l
            for i, l in enumerate(title)
            if i not in random.sample(range(len(title)), no_chars)
        ]
    )
    no_chars = random.randint(1, 2)
    developer = right.iloc[r, dpos]
    right.iloc[r, dpos] = "".join(
        [
            l
            for i, l in enumerate(developer)
            if i not in random.sample(range(len(developer)), no_chars)
        ]
    )

right.rename(columns={"developer": "dev"}, inplace=True)

Matches Data Set

The matching examples are passed to the matching models as pairs of indices. We do not need to provide non-matching examples. The models automatically pick non-matching examples from the Cartesian product of the left and right data frames as long as their indices are not in the matches data frame. For this application, the matches are constructed by the rows of the left and right data frames with the same index and the 3 duplicate matches we created.

matches = pd.DataFrame(
    {
        "left": left.index.append(right.iloc[-no_duplicates:,].index),
        "right": range(len(right)),
    }
)

Matching Model Setup

A matching model initialization requires instructions on constructing the similarity map between the left and right datasets. The instructions are passed to the model as a named list that specifies

1. which fields are used from each dataset and how they are associated with each other, and

2. how to encode them, i.e., how to calculate the similarity values.

If the column names of the left and right are the same, it suffices to supply the common name to the similarity map instruction list (see, e.g., the title, platform, and year items below). If two columns have different names, the instruction should have the form left_name~right_name, as it is, for instance, the case for the developing studio in this application (see item developer~dev). Not all columns of the datasets need to be used in the instructions. We do not use the scores and reviews columns in this example.

The model can be instructed to calculate one or more similarities for each column association. For instance, the instructions of this example specify two similarity calculations for the platform and year associations and one for the title and developer~dev. The neermatch provides a set of predefined similarity functions that can be used in similarity maps. The string similarities and ratios are calculated using RapidFuzz. The complete set of predefined functions can be retrieved by calling the available_similarities() function.

instructions = {
    "title": ["jaro_winkler"],
    "platform": ["levenshtein", "discrete"],
    "year": ["euclidean", "discrete"],
    "developer~dev": ["jaro"]
}

similarity_map = SimilarityMap(instructions)
print(similarity_map)

SimilarityMap[
  ('title', 'title', 'jaro_winkler')
  ('platform', 'platform', 'levenshtein')
  ('platform', 'platform', 'discrete')
  ('year', 'year', 'euclidean')
  ('year', 'year', 'discrete')
  ('developer', 'dev', 'jaro')]

A matching model object is constructed by passing the similarity map instructions. The model prepares encoding operations based on the passed instructions and uses them whenever the model is fitted or evaluated. We skip the construction details of the model in this example and refer the interested reader to the documentation entries of the DLMatchingModel, RecordPairNetwork, and FieldPairNetwork classes.

model = DLMatchingModel(similarity_map)

The model is compiled in the usual (keras) way. The compile function wraps the tensorflow.keras.Model.compile function, so all the functionality and options in the latter can be used here.

model.compile(
    loss = tf.keras.losses.BinaryCrossentropy(),
    optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001)
)

Matching Model Fit and Evaluation

The model is fitted using the fit function (see the fit documentation for details). The fit function extends the functionality of tensorflow.keras.Model.fit to accommodate the entity matching problem requirements. Firstly, instead of passing features and label arguments, fit expects the left, right, and matches data frames.

During training, counterexamples of matching records are automatically selected from the cross-product of left and right data frames based on the mismatch_share parameter. Including all the non-matching examples can lead to a highly unbalanced matching dataset. For each record in the left data set, appearing in k examples in the matches data set, there are up to n - k counterexamples, where n is the number of records in the right data set. The mismatch_share parameter controls the ratio of counterexamples to matches. For instance, if mismatch_share = 0.5, the encode function selects 50% of the possible counterexamples for each match. The counterexamples are selected randomly from the right dataset.

For instance, the left and right data sets in this example have 36 and 39 records, respectively, and the matches data set has 39 records. This allows us to construct 1365 non-matching examples. We set mismatch_share = 0.2, which means that for each matching example provided in matches, we get 7 non-matching examples (the integer part, i.e., the floor, of the number of counterexamples used).

The remaining arguments are similar to tensorflow.keras.Model.fit.

model.fit(
    left,
    right,
    matches,
    epochs=100,
    batch_size=32,
    verbose=0
)

The evaluate function overloads the tensorflow.keras.Model.evaluate function (see also the evaluate documentation). Similar to fitting a matching model, the evaluate call expects left, right, and matches data sets.

model.evaluate(left, right, matches, verbose = 0)

[............................................................] 1/88[=...........................................................] 2/88[==..........................................................] 3/88[==..........................................................] 4/88[===.........................................................] 5/88[====........................................................] 6/88[====........................................................] 7/88[=====.......................................................] 8/88[======......................................................] 9/88[======......................................................] 10/88[=======.....................................................] 11/88[========....................................................] 12/88[========....................................................] 13/88[=========...................................................] 14/88[==========..................................................] 15/88[==========..................................................] 16/88[===========.................................................] 17/88[============................................................] 18/88[============................................................] 19/88[=============...............................................] 20/88[==============..............................................] 21/88[===============.............................................] 22/88[===============.............................................] 23/88[================............................................] 24/88[=================...........................................] 25/88[=================...........................................] 26/88[==================..........................................] 27/88[===================.........................................] 28/88[===================.........................................] 29/88[====================........................................] 30/88[=====================.......................................] 31/88[=====================.......................................] 32/88[======================......................................] 33/88[=======================.....................................] 34/88[=======================.....................................] 35/88[========================....................................] 36/88[=========================...................................] 37/88[=========================...................................] 38/88[==========================..................................] 39/88[===========================.................................] 40/88[===========================.................................] 41/88[============================................................] 42/88[=============================...............................] 43/88[==============================..............................] 44/88[==============================..............................] 45/88[===============================.............................] 46/88[================================............................] 47/88[================================............................] 48/88[=================================...........................] 49/88[==================================..........................] 50/88[==================================..........................] 51/88[===================================.........................] 52/88[====================================........................] 53/88[====================================........................] 54/88[=====================================.......................] 55/88[======================================......................] 56/88[======================================......................] 57/88[=======================================.....................] 58/88[========================================....................] 59/88[========================================....................] 60/88[=========================================...................] 61/88[==========================================..................] 62/88[==========================================..................] 63/88[===========================================.................] 64/88[============================================................] 65/88[=============================================...............] 66/88[=============================================...............] 67/88[==============================================..............] 68/88[===============================================.............] 69/88[===============================================.............] 70/88[================================================............] 71/88[=================================================...........] 72/88[=================================================...........] 73/88[==================================================..........] 74/88[===================================================.........] 75/88[===================================================.........] 76/88[====================================================........] 77/88[=====================================================.......] 78/88[=====================================================.......] 79/88[======================================================......] 80/88[=======================================================.....] 81/88[=======================================================.....] 82/88[========================================================....] 83/88[=========================================================...] 84/88[=========================================================...] 85/88[==========================================================..] 86/88[===========================================================.] 87/88[============================================================] 88/88

{'TP': 39,
 'FP': 19,
 'TN': 1346,
 'FN': 0,
 'Loss': 62.71932601928711,
 'Accuracy': 0.9864672422409058,
 'Recall': 1.0,
 'Precision': 0.6724137663841248,
 'F1': 0.8041236996650696,
 'MCC': 0.8142813444137573}

Predictions and Suggestions

Matching predictions can be obtained in two ways from the fitted model. Either by calling predict or by calling suggest. The predict function returns a vector of prediction probabilities for each combination of left and right records. The prediction probabilities are stored in row-major order. First, the matching probabilities of the first row of left with all the rows of right are given. Then, the probabilities of the second row of left with all the rows of right are given, and so on. In total, the predict function returns a vector with rows equal to the product of the number of rows in the left and right data sets.

predictions = model.predict(left, right, verbose = 0)

fig, ax = plt.subplots()
counts, bins= np.histogram(predictions, bins = 100)
cdf = np.cumsum(counts)/np.sum(counts)
ax.plot(bins[1:], cdf)
ax.set_xlabel("Matching Prediction")
ax.set_ylabel("Cumulative Density")
plt.show()

The suggest function returns the best matching predictions of the model for each row of the left dataset. The prediction probabilities of predict are grouped by the indices of the left dataset and sorted in descending order. The caller can choose the number of returned suggestions by setting the count argument of suggest.

suggestions = model.suggest(left, right, count = 3, verbose = 0)
suggestions["true_match"] = suggestions.loc[:, ["left", "right"]].apply(
    lambda x: any((x.left == matches.left) & (x.right==matches.right)), axis=1
)
suggestions = suggestions.join(
    matches.iloc[-no_duplicates:,:].assign(duplicate=True).set_index(['left', 'right']),
    on = ["left", "right"],
    how = "left"
)
suggestions.duplicate = suggestions.duplicate.apply(
    lambda x: False if pd.isna(x) else x
)
suggestions = suggestions.sort_values(
    by=["left", "prediction"], ascending=[True, False]
)

suggestions

	left	right	prediction	true_match	duplicate
0	0	0	9.778342e-01	True	False
1	0	14	9.169472e-01	False	False
2	0	4	9.598946e-04	False	False
3	1	1	9.718049e-01	True	False
4	1	23	8.463337e-05	False	False
5	1	10	7.474281e-05	False	False
6	2	2	9.787740e-01	True	False
7	2	7	1.013026e-03	False	False
8	2	35	6.155510e-04	False	False
9	3	3	9.766330e-01	True	False
10	3	30	1.074756e-03	False	False
11	3	21	5.062067e-05	False	False
12	4	4	9.787740e-01	True	False
13	4	12	9.383137e-01	False	False
14	4	0	9.753937e-04	False	False
15	5	5	9.777580e-01	True	False
16	5	15	7.423223e-05	False	False
17	5	27	3.159808e-05	False	False
18	6	6	9.773185e-01	True	False
19	6	24	2.507875e-05	False	False
20	6	17	7.486212e-08	False	False
21	7	7	9.781616e-01	True	False
22	7	2	1.006788e-03	False	False
23	7	35	6.105449e-04	False	False
24	8	8	9.772826e-01	True	False
25	8	9	3.877738e-05	False	False
26	8	23	1.128767e-07	False	False
27	9	9	9.716737e-01	True	False
28	9	17	5.324303e-04	False	False
29	9	28	4.801422e-04	False	False
30	10	10	9.733325e-01	True	False
31	10	23	9.662850e-01	False	False
32	10	1	7.031505e-05	False	False
33	11	11	9.782392e-01	True	False
34	11	29	4.065759e-04	False	False
35	11	19	5.362284e-05	False	False
36	12	12	9.742600e-01	True	False
37	12	4	9.637859e-01	False	False
38	12	22	1.027538e-03	False	False
39	13	13	9.783336e-01	True	False
40	13	37	9.752246e-01	True	True
41	13	33	9.751084e-01	False	False
42	14	14	9.712535e-01	True	False
43	14	0	9.653772e-01	False	False
44	14	4	1.020161e-03	False	False
45	15	15	9.767061e-01	True	False
46	15	24	3.229330e-04	False	False
47	15	5	7.657281e-05	False	False
48	16	16	9.648299e-01	True	False
49	16	30	4.539440e-05	False	False
50	16	34	3.271291e-05	False	False
51	17	17	9.758328e-01	True	False
52	17	21	1.238351e-03	False	False
53	17	9	5.116409e-04	False	False
54	18	18	9.786937e-01	True	False
55	18	35	9.030289e-01	False	False
56	18	36	8.958049e-01	False	False
57	19	19	9.778247e-01	True	False
58	19	33	1.464835e-03	False	False
59	19	13	1.276108e-03	False	False
60	20	20	9.783897e-01	True	False
61	20	32	9.765700e-01	False	False
62	20	37	3.983492e-04	False	False
63	21	21	9.739489e-01	True	False
64	21	17	1.318118e-03	False	False
65	21	9	4.996893e-04	False	False
66	22	22	9.768478e-01	True	False
67	22	12	7.845204e-04	False	False
68	22	4	5.201011e-04	False	False
69	23	23	9.778675e-01	True	False
70	23	10	9.233682e-01	False	False
71	23	1	7.060236e-05	False	False
72	24	24	9.670053e-01	True	False
73	24	15	3.808707e-04	False	False
74	24	6	3.039734e-05	False	False
75	25	25	9.785444e-01	True	False
76	25	13	9.746334e-01	False	False
77	25	33	9.737168e-01	False	False
78	26	26	9.766273e-01	True	False
79	26	38	9.702485e-01	True	True
80	26	31	4.269315e-04	False	False
81	27	27	9.779201e-01	True	False
82	27	15	3.216346e-05	False	False
83	27	5	3.040044e-05	False	False
84	28	28	9.773872e-01	True	False
85	28	17	5.012505e-04	False	False
86	28	21	5.000369e-04	False	False
87	29	29	9.787740e-01	True	False
88	29	11	4.129025e-04	False	False
89	29	32	8.954816e-05	False	False
90	30	30	9.761566e-01	True	False
91	30	3	1.078105e-03	False	False
92	30	17	7.555061e-05	False	False
93	31	31	9.777137e-01	True	False
94	31	26	5.508639e-04	False	False
95	31	38	3.956477e-04	False	False
96	32	32	9.785187e-01	True	False
97	32	20	9.764295e-01	False	False
98	32	37	3.983492e-04	False	False
99	33	33	9.786212e-01	True	False
100	33	13	9.748630e-01	False	False
101	33	25	9.736227e-01	False	False
102	34	34	9.765700e-01	True	False
103	34	30	4.525233e-05	False	False
104	34	17	4.510958e-05	False	False
105	35	36	9.575554e-01	True	True
106	35	35	9.503515e-01	True	False
107	35	18	9.235189e-01	False	False