Skip to contents

Changelog

Source: NEWS.md

OccurrenceCleaner 0.2.0

  • flag_occurrences(n_threads = ) runs the exact outlier test and the proximity and point-in-polygon queries on several threads (OpenMP). The flags do not depend on the number of threads.
  • A streamed run reports its progress and an estimate of the time left, for the current step and for the whole run, in out/progress.txt and, with verbose, on the console every minute.
  • The issue tests read the issue column of a chunk in one vectorised pass.
  • keep_unflagged() returns the records of a flagged data frame that no test of a rule flagged, by default every test that ran, keeping the flag columns and attributes.
  • Each test has a cleaning function, clean_<test>() (clean_capital(), clean_outlier(), …), that runs the test and returns the rows of a data frame it does not flag, so tests can be chained in a pipe. Its arguments are the test’s buffer or options with their defaults, and it is documented on the test’s help page. The functions are generated from the test registry and run flag_occurrences(); a chain applies each test to the records the previous steps kept.
  • Without tests, flag_occurrences() runs every test whose inputs are available: the columns each test reads, and range polygons for range. Running a test only adds flags; which records to exclude stays a cleaning rule chosen afterwards. default_tests() is removed; name CoordinateCleaner’s set explicitly when it is wanted.
  • test_plan() lists, for an input, the tests that would run and why each other test would not, and for flag output the tests that ran and why the others did not. Flagged data frames (attribute occ_test_plan) and on-disk output (flags_meta.rds) record it, and print() names the tests that could not run.
  • gentiana: the 33,025 GBIF records of the genus Gentiana in Austria under CC0 or CC BY 4.0 from the snapshot of 2026-10-01, for examples and the case study of the package paper.
  • print() methods for occ_flags (a header and the first records, with the tests each failed), occ_tally (records removed and kept per rule) and occ_columns; occ_flagged prints its reference on one line. summary() stays the full per-test report.
  • has_flag() takes a flagged data frame as well as the flag words.
  • flag_reference() returns the reference label recorded with a set of flags, in memory or on disk.
  • A rule that names a test the flags were not computed with is an error in has_flag() on a flagged data frame, tally_flags(), summary() and plot(), instead of removing nothing for that test.
  • Points of a user gazetteer with missing or out-of-range coordinates give a warning that says how many are not used.
  • plot() keeps a label on every bar however many tests a rule holds; the map takes xlim and ylim.
  • The institution gazetteer holds every institution of CoordinateCleaner’s table with coordinates but one, an expert group listed in Antarctica (23,895 points, from 18,245). It adds the 1,062 herbaria of Index Herbariorum that GRSciColl does not place and the 2,468 publishing organisations of the GBIF registry with coordinates. The 2,120 further points place the institutions of CoordinateCleaner’s list that no source holds under their name, from Wikidata, OpenStreetMap (Nominatim) or GeoNames, choosing among candidates by CoordinateCleaner’s point (source “CoordinateCleaner list”). For 496 institutions no source places the institution itself, and the point is the town of the country nearest CoordinateCleaner’s point.

Streamed input

  • duplicate compares each record with every record before it in the whole input, on disk as in a data frame: streamed input gets the same flags as the same records in one data frame, in file and row order. Earlier versions compared records within one chunk at a time, so copies in different chunks or files were missed. duplicate is now a grouped test: it buckets a 124-bit hash of each record’s key on disk and needs a path rather than a single-run vectra_node.
  • A streamed run records its configuration in out/flags_config.rds before writing any output (tests, columns, keep, reference, buffers, settings, duplicate options, package version, and each input file’s size and modification time); a later call into the same out must match it, or it stops and names what differs. Earlier versions checked only the tests, layout and reference, and only after a run had finished.
  • The group stage checkpoints its state in out/.group after a file or a bucket once five minutes have passed and at the end of each pass, so an interrupted run resumes from the last checkpoint. Its hits are stored on disk by file and block of rows, and the record pass reads one block at a time; the bucket count grows with the input, and a bucket too large for memory is split further.
  • Files on disk are read with only the columns the tests, keep and duplicate_by use, in the record pass and in the group pass.
  • The group pass counts species with summed per-chunk tables, tallies large species with one split per chunk, holds bucket rows in memory and writes them in batches, and computes the exact outlier distances once per pair. On three GBIF snapshot files (1.9 million records) the group pass takes 101 s instead of 344 s.
  • A vectra_node given as input runs once, so the grouped tests, which read the input more than once, stop with a message asking for the path instead.
  • dataset_rounded no longer warns when a dataset has no regular spacing between 0.01 and 2 degrees.
  • print() and summary() of a flagged data frame or tally subset to other columns fall back to the data frame methods.

OccurrenceCleaner 0.1.0

Initial release.

Flagging

  • flag_occurrences(): one verb for a data frame or an on-disk table (Parquet directory, .vtr, delimited file, or any vectra_node). Thirty tests, one bit each in an integer flag word; nothing is dropped.
  • Twelve tests reproduce CoordinateCleaner’s record tests with its default buffers (invalid, zero, equal, capital, centroid, institution, gbif_headquarters, sea, urban, country_mismatch, duplicate, and the fossil age test age_equal, as cf_equal()); eight more are CoordinateCleaner’s grouped and reference tests: outlier (cc_outl()), age_outlier (cf_age()), spatiotemporal_outlier (cf_outl()), age_range_outlier (cf_range()), dataset_ddmm (cd_ddmm()), dataset_rounded (cd_round()), range (cc_iucn(), with user range polygons) and aohi (cc_aohi()); eight read GBIF fields (issue_coordinate, issue_taxon, issue_date, issue_rounded, basis_fossil, basis_living, establishment_introduced, establishment_managed); two set the land test against the habitat of the record’s genus (sea_nonmarine, land_marine). Tests read from a record field carry that field as a prefix.
  • Proximity tests run as kd-tree candidate searches on the unit sphere decided by WGS84 geodesic distance; polygon tests as grid-indexed point-in-polygon with exact boundary cells. Both in C++ with no external library.
  • Streamed input is processed per file, written as .vtr parts with the kept columns, and resumable.
  • The grouped tests (outlier, the fossil outlier tests, the dataset tests) run in a group pass over the whole input before the record tests; on streamed input it reads every file, buckets small groups on disk and keeps its result for a resumed run. Their options, with CoordinateCleaner’s defaults, go in settings. outlier flags the records of outlier cells on the raster approximation for large species, where CoordinateCleaner 3.0.1 flags none; the fossil tests seed their age draws per taxon.

Rules and tallies

  • tally_flags(): loss tables by grouping columns, with numeric bins and named rules, over a data frame or the on-disk output. Per rule it also counts the records each test is the sole reason for.
  • summary() on flagged records, on-disk output or a tally gives the percentage of records each test and rule removes, with the sole-reason share; plot() draws it as bars or, by group, as a grid, and maps kept and removed records with type = "map".
  • has_flag(), flag_mask(), flag_tests(), default_tests().
  • as_spatialvalid() writes flagged records, in memory or on disk, in the layout of CoordinateCleaner’s spatialvalid object (.cap, .zer, …, .summary, TRUE for pass), so code and methods written for it run unchanged; flag_tests() gives the column name per test.

Reference data

  • Own gazetteers rebuilt from Natural Earth (province centroids, land at 1:110 million and 1:50 million, urban areas, countries), the CIA World Factbook (capitals: 232 points, of which CoordinateCleaner’s capitals are a rounding for 221 of 225, plus 243 Natural Earth capitals, alternative capitals and dependent-territory capitals marked extension), the thematicmapping.org World Borders Dataset 0.3 and the GEOLocate service (country centroids: 1,069 points from the three sources of CoordinateCleaner’s table, which hold 645 of its 660 country centroids exactly, plus 516 Natural Earth polygon centroids and label points marked extension), GBIF GRSciColl, Wikidata, GeoNames and BGCI (institutions: 6,080 GRSciColl entries without the five the zero test flags; 7,810 Wikidata botanical gardens, natural history museums, zoos, aquaria and herbaria; 700 museums of Wikipedia’s lists of natural history museums (English and six other language editions) placed by the town or institution the list names; the three GeoNames places of CoordinateCleaner’s table; 3,652 gardens of BGCI GardenSearch with BGCI’s own coordinates; column extension marks the 7,665 points from sources CoordinateCleaner’s table does not draw on), each pinned by version and SHA-256 in reference_info(); the 1:10 million land layer is fetched on demand by download_reference().
  • The Artificial Hotspot Occurrence Inventory (Park et al. 2023) is built from its Dryad deposit (CC0) for the aohi test and returned by reference_points("aohi").
  • reference = "CoordinateCleaner" reads that package’s tables at run time and reproduces its flags; the equivalence suite documents every difference.
  • gazetteers in flag_occurrences() replaces the reference layer of single tests with user points (data frame) or polygons (sf/sfc), as the ref arguments of CoordinateCleaner’s functions; the layer’s size and MD5 checksum are recorded in the reference label.
  • export_pyrate() writes fossil records as PyRate input files, the layout of CoordinateCleaner’s write_pyrate().
  • The outlier test’s sampling correction (settings$outlier$sampling_thresh) reads records per country from gazetteers = list(sampling = ...) instead of querying GBIF at run time.
  • reference_points() returns the points of a proximity test with their attributes, to inspect, filter or extend.
  • Genus habitats for sea_nonmarine and land_marine are built from the Interim Register of Marine and Nonmarine Genera (Darwin Core archive of 2026-10-06, CC BY 4.0) and returned by reference_habitat(); gazetteers = list(habitat = ...) replaces them.