← SASnRHandling duplicates

Introduction

Duplicate rows come up in almost every real dataset — a subject entered twice, a lab value repeated across two sources, a merge that accidentally widened the data.
Two common asks: keep one row per key combination (nodupkey in SAS), or remove rows that are exact copies across every column (nodup / noduprec in SAS).
In SAS, proc sort nodupkey keeps one row per BY-group; proc sort noduprec drops full-row duplicates when paired with by _all_.
Tidyverse: distinct() for full-row deduplication; group_by() %>% slice(1) for one row per key combination.
Base R: !duplicated(df) for full rows; !duplicated(df[c("k1","k2")]) for a key-based version after sorting.
This lesson works two small AE datasets — one where duplication is key-based and one where full rows repeat — so we can see both patterns side by side.

Handling duplicates

SAS (Base SAS)+

R (tidyverse)+

library(tidyverse)

ae <- tribble(
  ~subjid, ~term,       ~stdtc,
  "1001",  "Headache",  "2010-01-05",
  "1001",  "Headache",  "2010-01-08",
  "1001",  "Nausea",    "2010-01-08",
  "1002",  "Headache",  "2010-02-15"
)

ae1 <- tribble(
  ~subjid, ~term,       ~stdtc,
  "1001",  "Headache",  "2010-01-05",
  "1001",  "Headache",  "2010-01-08",
  "1001",  "Nausea",    "2010-01-08",
  "1002",  "Headache",  "2010-02-15",
  "1002",  "Headache",  "2010-02-15"
)

#---------------------------------------
# Duplicate based on key variables (nodupkey)
#---------------------------------------

nodupkey <- ae %>%
  arrange(subjid, term, stdtc) %>%
  group_by(subjid, term) %>%
  slice(1)

#---------------------------------------
# Duplicate record (noduprec / nodup)
#---------------------------------------

noduprec <- ae1 %>%
  distinct()

Tidyverse splits the job into two clear calls: distinct() for full-row deduplication and group_by() %>% slice(1) for key-based deduplication.
ae %>% arrange(subjid, term, stdtc) %>% group_by(subjid, term) %>% slice(1) is the direct tidyverse equivalent of proc sort nodupkey by subjid term.
The arrange() step controls which row survives within each key combination — typically we sort so the earliest or most informative row lands first.
slice(1) keeps just the first row per group; switch to slice(n()) to keep the last row per group, equivalent to picking the latest record.
ae1 %>% distinct() drops rows that are exact copies across every column — the tidyverse counterpart of proc sort noduprec.
distinct() accepts a variable list too: distinct(subjid, term, .keep_all = TRUE) keeps the first row per key and retains the rest of the columns.
We run and compare the counts in NODUPKEY and NODUPREC against the SAS output — they should line up row for row.

Happy learning!

Search lessons

Introduction

Handling duplicates