Flags every record with the same longitude, latitude and species as an
earlier record; the first of each set is not flagged. duplicate_by adds
columns that must also match; duplicate_cell bins coordinates to a grid
of that many degrees first. A missing species counts as one value, as in
duplicated(), so records without a species at the same coordinates are
duplicates of each other.
Arguments
- x
A data frame of occurrence records.
- by
Additional column names a duplicate must share, as
duplicate_byinflag_occurrences().- cell
Cell size in degrees for comparing coordinates, as
duplicate_cellinflag_occurrences().- ...
Further arguments to
flag_occurrences():columns,reference,gazetteers,land_scale.
Value
clean_<test>(): the rows of x the test does not flag, with the
columns of x unchanged. See keep_unflagged(), Cleaning functions.
Details
The test compares each record with all records before it in the input
order: the rows of a data frame, or on streamed input the files in sorted
order and the rows within each file. Streamed input therefore gets the same
flags as the same records in one data frame. It is a grouped test: the
records are bucketed by a 124-bit hash of their key in a pass before the
record tests, so it needs a path rather than a single-run vectra_node.
The probability that any two different keys share a hash is about 2e-18
for ten billion records.
CoordinateCleaner
Equivalent to cc_dupl(), with duplicate_by
as its additions. Records with invalid coordinates are not flagged here;
cc_dupl() flags the second of two such records.
See also
flag_occurrences(), flag_tests()
Other tests:
clean_age_equal(),
clean_age_outlier(),
clean_age_range_outlier(),
clean_aohi(),
clean_basis_fossil(),
clean_capital(),
clean_centroid(),
clean_country_mismatch(),
clean_dataset_ddmm(),
clean_dataset_rounded(),
clean_equal(),
clean_establishment_introduced(),
clean_gbif_headquarters(),
clean_institution(),
clean_invalid(),
clean_issue_coordinate(),
clean_outlier(),
clean_range(),
clean_sea(),
clean_sea_nonmarine(),
clean_spatiotemporal_outlier(),
clean_urban(),
clean_zero()
Examples
x <- data.frame(species = "a", decimalLongitude = c(16.37, 16.37, 16.3701),
decimalLatitude = 48.21, eventDate = c("2020", "2021", "2020"))
flag_occurrences(x, tests = "duplicate")$flag_duplicate
flag_occurrences(x, tests = "duplicate", duplicate_by = "eventDate")$flag_duplicate
flag_occurrences(x, tests = "duplicate", duplicate_cell = 0.01)$flag_duplicate
clean_duplicate(x, by = "eventDate")