pythonrevaluation-metrics

A-Eye

Ben Gorman

Ben Gorman

Life's a garden. Dig it.

You're building an image search engine called A-eye 👁️. You're using Mean Average Precision at K (MAP@K) to evaluate the performance of your results.

Your dataset contains three queries and their predicted relevance on 1000 images. Calculate MAP@5 for this data.

rng = np.random.default_rng(2703)
df = pd.DataFrame({
    'id': np.tile(range(1000), reps=3),
    'query': np.repeat(["dolphin", "emu", "gorilla"], repeats=1000),
    'pred_score': rng.uniform(low=0, high=1, size=3000),
})
 
df["relevant"] = False
df.loc[(df.id.isin(rng.choice(1000, size=25, replace=False))) & (df["query"] == "dolphin"), "relevant"] = True
df.loc[(df.id.isin(rng.choice(1000, size=25, replace=False))) & (df["query"] == "emu"), "relevant"] = True
df.loc[(df.id.isin(rng.choice(1000, size=25, replace=False))) & (df["query"] == "gorilla"), "relevant"] = True
 
print(df)
#        id    query  pred_score  relevant
# 0       0  dolphin    0.690034     False
# 1       1  dolphin    0.555039     False
# 2       2  dolphin    0.001131     False
# 3       3  dolphin    0.453131     False
# 4       4  dolphin    0.942450     False
#    ...      ...         ...       ...
# 2995  995  gorilla    0.650423     False
# 2996  996  gorilla    0.883602     False
# 2997  997  gorilla    0.756802     False
# 2998  998  gorilla    0.143045     False
# 2999  999  gorilla    0.944064     False
library(data.table)
 
set.seed(6186)
dt <- data.table(
    id = rep(1:1000, times = 3),
    query = rep(c("dolphin", "gorilla", "emu"), each = 1000),
    pred_score = runif(3000, min = 0, max = 1)
)
 
dt[, relevant := FALSE]
dt[id %in% sample(1000, size = 25, replace = FALSE) & query == "dolphin", relevant := TRUE]
dt[id %in% sample(1000, size = 25, replace = FALSE) & query == "gorilla", relevant := TRUE]
dt[id %in% sample(1000, size = 25, replace = FALSE) & query == "emu", relevant := TRUE]
 
print(dt)
#         id   query pred_score relevant
#    1:    1 dolphin  0.3276352    FALSE
#    2:    2 dolphin  0.5503843    FALSE
#    3:    3 dolphin  0.2108938    FALSE
#    4:    4 dolphin  0.3693824    FALSE
#    5:    5 dolphin  0.5566705    FALSE
#   ---                                 
# 2996:  996 gorilla  0.7887701    FALSE
# 2997:  997 gorilla  0.1574011    FALSE
# 2998:  998 gorilla  0.2992015    FALSE
# 2999:  999 gorilla  0.3836474    FALSE
# 3000: 1000 gorilla  0.6554015     TRUE

Solution

This content is gated

Subscribe to the product below to gain access