get_config_log() function to retrieve configuration log entries from the config_log table.
ss), task name (task), and date range (start_date, end_date).data.table with the filtered entries.update_config_log() to also route custom messages (...) to the message() function for clearer console output.update_config_log function. Logs configuration updates including surveillance system (ss), task name (task), and a custom message.CS9_DBCONFIG_ROLE_CREATE_TABLE in environmental variables.as.character(lubridate::now()) adds microseconds, which breaks the SQL upload. This is now replaced by cstime::now_c().update_config_tasks_statsDBPartitionedTableExtended_v9$info() bug fixed with argument collapse=TRUE.confirm_indexes in DBPartitionedTableExtended_v9.DBPartitionedTableExtended_v9$nrow() now has a new argument collapse=FALSE that provides partion-specific resultsDBPartitionedTableExtended_v9$info() now includes sizes in MBDBPartitionedTableExtended_v9$nrow()DBPartitionedTableExtended_v9 so that it is easier to use multiple partitioning variables.partitions_randomized in DBPartitionedTableExtended_v9 so that when running in parallel, the database tables don't get locked.remove_table in DBPartitionedTableExtended_v9get_config_tasks_statsDBTableExtended_v9 now automatically includes a column for all tables, called auto_last_updated_datetime, which is automatically calculated each time that row is changed.DBPartitionedTableExtended_v9, which allows for one dataset to be partitioned amongst multiple SQL tables automatically.SurveillanceSystem_v9 constructor now takes in an argument called implementation_version, which can be used to identify what version of analytics code is currently being run.update_config_last_updated has now been replaced by update_config_tables_last_updated (which contains when the tables were last updated) and config_tasks_stats (which contains all the runtimes of the tasks).SurveillanceSystem_v9 now uses an internal R6 class DBTableExtended_v9 (which extends csdb::DBTable_v9) instead of using csdb::DBTable_v9 directly. DBTableExtended_v9 calls update_config_last_updated after altering a database table.TRUNCATE when no row of it must stay. It does so also before it adds cs9_stg_id to an old staging table. Otherwise the fence deletes the rows of dead runs in transactions of at most 200,000 rows. Before, a dead run of about 25 million rows meant two transactions of that size at the next run.cs9_stg_id. On 2026-10-16 PostgreSQL went down twice during a publish of about 25 million rows in one transaction. A recorded delete runs in one transaction per partition, and drop_all_rows() becomes a TRUNCATE. A reader can see a partly published run. After a publish that stopped part-way, a retry with the same run_key checks the plan list, runs no plan and publishes the rest. A run of the task with a new run_key first publishes the rest of that run. A staging table from before this version gets cs9_stg_id in place and keeps its rows. SQLite publishes in one transaction, as before. cs9 needs csdb 2026.10.18.CS9_DBCONFIG_PG_LOAD, and it gives csdb no pg_load. csdb 2026.10.17, which cs9 now needs, sends every PostgreSQL load as JSON over the open connection.CS9_DBCONFIG_PG_LOAD sets pg_load in every database configuration and every csdb connection that cs9 builds. Unset or empty is psql. With json, csdb sends the rows of a load over the open connection, and a load makes no psql login. This needs csdb 2026.10.16.library(cs9) and reload_db_config() with an error that names the variable and the allowed values. check_environment_setup() shows the value. Other drivers ignore it.Task$run() forks its workers once: cores workers, or one per middle plan when there are fewer. Each worker runs middle plans until none is left. Before, each middle plan had its own fork.INSERT statement over the connection of the table, on PostgreSQL and SQLite. Before, each write on PostgreSQL ran psql, which is one more login.INSERT ... ON CONFLICT on its keys. It needs the primary key that csdb creates with the table.config_data_hash_for_each_plan open from one plan to the next. Outside a worker each call closes it, as before.Task$run() sets the csdb context to <task>|parent, and each plan sets <task>|<plan>. PostgreSQL shows it as the application_name of each connection. An open connection gets the name by the rule of csdb: the characters A-Z a-z 0-9 _ | . : -, at most 63. This needs csdb 2026.10.15.run_key keeps the date of its first attempt also when its task has no tables, or only config tables. This holds on PostgreSQL and SQLite. run_today() and argset$today give that date on a retry with the key. Before, a retry on a later day got the date of the retry.cs9_run in the config database. It takes no lock and stages nothing, and each attempt runs every plan.run_today() reads the clock, as before.run_key, the publish sends one DELETE per live table for each 500 operations. It joins their conditions by OR. Before, it sent one DELETE per operation and live table.DELETE without a condition.drop_rows_where() or keep_rows_where() MUST test only the row itself, with no subquery on the same table. In a keyed run every condition reads the live table as it stood before the publish.cs9_task and cs9_run_key, then the partition column of a partitioned table. A keyed run adds it to a staging table that exists when the run begins. The index is verified on SQLite and PostgreSQL, not on SQL Server.norsyss_data_import_consultations recorded 705 operations on a table of 106 partitions. Its publish ran for over 45 minutes.config$tables. Before, cs9 counted every table in the database and schema of the config access as a config table.run_key staged nothing and wrote its tables directly.run_key whose task has tables but stages none of them stops with an error. A task whose every table is a config table, and a task without tables, still run unstaged.run_task_detached(), run_task_sequentially_as_callr_bg_using_load_all(), run_task_sequentially_as_rstudio_job_using_load_all() and TaskJob$new() take run_key = NULL. They pass it to run_task() in the child process, so a developer can start a staged run.run_task(), or a command-line argument of the detached Rscript. run-task.sh takes --key.run-task.sh takes --rscript. run_task_detached() passes the Rscript of the calling R by its full path, so the child runs the same R, also under R CMD check.run_key runs the last plan again, like plan 1. A check in the last plan therefore runs again after a publish that failed.add_table(), add_partitionedtable() and the cs9 table classes refuse a table name that ends in _cs9stg. Such names are reserved for staging.run_today() is exported. It returns the date of the first attempt of the run with a run_key in progress, or today's date outside such a run.run_key, which a scheduler can pass, stages its writes to the task's own tables. It publishes them in one transaction after the last plan. A failed run leaves the task's tables unchanged.run_task() and Task$run() take run_key = NULL by default, and a run without a key touches no staging table.run_key skips each plan that the first attempt finished, except plan 1.Task$run() with a run_key builds the plans again on every attempt, as run_task() does.run_key, tbl() and nrow() read the published table. staged_tbl() reads the rows that the run staged.run_key stops before any plan runs when its plans differ from the first attempt. The order of the plans counts.run_key does nothing, also after a run with another key.run_key that cannot stage its writes stops with an error, before any write. Two tables with one name are such a case, and so is a table that cs9 cannot stage. A table of the config access is never staged.staged_tbl() stops the run, because plans MAY read published tables only.run_key, argset$today is the date of the first attempt.config_tables_last_updated gets one row per changed table, once per run with a run_key, at the publish.run_key stops while the first is alive. Each such run holds one more connection, for its lock and its publish.<table>_cs9stg. The tables cs9_run, cs9_run_plan_list, cs9_run_plan_done and cs9_run_operation hold the state of each run.vignette("rebuilding-a-table-safely") shows what a reader sees during a run, a failed run and its retry.insert_data() method of a partitioned table loads every partition with one call to csdb::insert_data_many(). On PostgreSQL an insert needs one psql login for the load, plus one for the last-updated bookkeeping, not one for each partition. With SQLite or SQL Server, csdb still loads one partition at a time.confirm_insert_via_nrow = FALSE), an insert that fails stores no row in any partition. The exception is an error that says the connection broke, or that a row count differed after psql finished. The rows may then be stored, and the error says to count them. With confirm_insert_via_nrow = TRUE, a duplicate key makes csdb insert partition by partition, so a failure can leave some partitions stored.insert_data() gives a warning that names the partitions and the error, and returns normally.insert_data() that fails writes no row of its own to config_tables_last_updated. Before, it recorded the partitions that it loaded before the failure. With confirm_insert_via_nrow = TRUE, or with SQLite or SQL Server, each partition that loaded before the failure still records itself.vignette("how-a-task-runs") says what a failed insert into a partitioned table stores, and when to count the rows.Task$run() runs each parallel plan once. Before, a failing plan ran up to 5 times, and only the error of the last try reached the log. The scheduler, for example Airflow, retries the whole task.Task$run() sets config$plan_attempt_index to 1 for every plan. It stays for callers that read it.pbmcapply::pbmclapply() had scheduled on the same core. Task$run() then stops with one error that counts the failed plans and holds the message of each, in plan order. Before, it held the message of the first failed plan only.check_environment_setup() no longer requires CS9_DBCONFIG_PASSWORD for PostgreSQL Unicode, so CS9 loads when a hook from csdb::csdb_set_password_hook() supplies the password. An unset password is still an empty string.vignette("backends") and vignette("installation") say when CS9_DBCONFIG_PASSWORD is optional.add_task() accepts cores as a function with no arguments. Task$run() calls it each time the task runs, so a cap that an environment variable sets after the package loads takes effect. A number works as before.TaskJob and run_task_sequentially_as_rstudio_job_using_load_all() still run a task on 1 core when its cores is a function.run_task() builds the plans of a for_each_plan task again on every call, so argset$today and argset$yesterday are the date of the run. Before, they were the date when the package loaded.get_task() and the shortcut_get_*() methods still show the plans of the last build, with the date of that build.vignette("how-a-task-runs") shows a task that reads its cores from an environment variable that changes after add_task().#rule-plans-per-run of vignette("how-a-task-runs") states when a task builds its plans, and how to run plans that you edit by hand.vignette("task-shapes") holds all seven task shapes. The sections of the removed vignette("more-task-shapes") keep their heading ids there.vignette("task-shapes") has a diagram for each shape, and vignette("how-a-task-runs") has a diagram of one run. Each diagram comes from a run that the vignette records and checks when it builds. ggplot2 is in Suggests for these diagrams.insert_data(), upsert_data(), drop_all_rows_and_then_insert_data() and drop_all_rows_and_then_upsert_data() write their config_tables_last_updated rows in one upsert. The rows are the same: one per completed partition.DBTableExtended_v9 method writes at most one config_tables_last_updated row. Before, drop_all_rows_and_then_upsert_data() wrote up to 3 rows for one table, and an insert_data() that fell back to an upsert wrote 2.get_config_log() help page says that start_date and end_date filter the date column. It named a timestamp field that does not exist.get_config_log() filters by ss and by task. Before, both filters stopped with first argument has length > 1.update_config_tasks_stats() stores the start and stop datetimes with milliseconds. update_config_data_hash_for_each_plan() stores its datetime with milliseconds. So two runs in one second keep two rows. The table schema does not change.DATETIME rounds to about 3 ms.drop_all_rows(), drop_rows_where() and keep_rows_where() write their config_tables_last_updated rows in one upsert. After an error, that upsert includes the partitions that completed before the error.config_tables_last_updated row per partition, in separate writes.cyclocomp_linter: R/2_onLoad.R, R/r6_DBPartitionedTableExtended_v9.R, R/r6_Task.R and R/r6_TaskJob.R. The complexity check skips all code in those files, which includes the whole R6 classes DBPartitionedTableExtended_v9, Task and TaskJob.Task$run() reads and stores the data hash of the last plan under its own
index_plan. With 4 or more plans, cores > 1 and a non-interactive session,
it used index_plan 1, the index of the first plan.DBTableExtended_v9 accept load_timeout and pass
it to csdb. csdb 2026.10.4 calls insert_data() and upsert_data() with this
argument, which stopped with unused argument. cs9 now needs csdb >= 2026.10.4.vignette("task-shapes") shows two task shapes that publish a run only when
the whole run succeeds. Both shapes run against SQLite when the vignette builds.update_config_log() writes datetime in milliseconds, at least 4 ms after
its previous row in the R process. Two runs of one task in the same second
failed on SQLite with UNIQUE constraint failed. The table schema does not
change.
Two separate R processes can still collide in the same millisecond, and the
local time repeats at the autumn daylight-saving change, as it did before.
Verified on PostgreSQL 16 and SQL Server 2022, where datetime rounds to
1/300 s and the 4 ms step keeps rows distinct.run_task() builds the plans of a task with a plan analysis on every call. A
second run_task() in one session reused the plans of the first. The
shortcut_get_*() methods still reuse the plans of the last build.vignettes/creating-a-task.Rmd.orig no longer carries an empty
if (requireNamespace("magrittr", ...)) { } block. Dropping library(magrittr)
left the guard behind with nothing in it. The .orig file ships in the tarball
and _PRECOMPILER.R regenerates the vignette from it.The package drops magrittr. Every %>% is now the base pipe |>, and
magrittr is gone from DESCRIPTION.
The rewrite is a relocation, not an edit. Each %>% call was transformed the
way R's parser transforms |>, and the resulting tree was required to match
the tree parsed from the rewritten file. A file whose trees disagreed was left
untouched and converted by hand instead.
%>% is no longer exported. R/x_rexport.R existed only to re-export it and
is deleted, and the "Imported Functions" section of _pkgdown.yml goes with
it. Code that relied on library(cs9) for the pipe must use |> or attach
magrittr itself.
vignettes/creating-a-task.Rmd.orig and vignettes/file-layout.Rmd.orig are
converted too. _PRECOMPILER.R regenerates the shipped .Rmd from them, so
converting only the .Rmd would have been undone on the next precompile.
The version runs one day ahead of the calendar on purpose. r-universe already published today's number from a different commit, and one version must never name two trees.
R/zzz_imports.R names callr, csutil, later, pbmcapply and
progress in one plain function, which clears the declared-but-unused note.
R CMD check does not walk the public = list(...) of an R6Class, so a
pkg::fn() call inside an R6 method is invisible to its dependency scan.progress looked dead, because it is never written as
progress:: anywhere in this package. It is required at load time all the
same: .onLoad calls progressr::handler_progress(), and that function's
body calls progress::progress_bar. Removing it made the package fail to
install.run_task_detached() and inst/scripts/run-task.sh run one task in a process that outlives the R session which started it. This is the non-interactive counterpart to run_task_sequentially_as_callr_bg_using_load_all(). Use that one in Positron, where a person watches a live console, and this one when nobody is watching: a script, a cron entry, or an agent driving a machine over SSH. The two are not interchangeable. TaskJob passes supervise = TRUE, so callr kills the task when the parent R process exits, and it streams output through later::later(), which needs an event loop that Rscript does not drive.kill -0 separates "running" from "died without reporting".global$ss.norsyss_data_import_consultations exited 0 with 212 rejected COPY statements, one per partition for each of two weeks whose rows already existed. load_data_infile.db_postgres runs psql through system2() and never reads its exit status, so a rejected COPY does not reach R at all. A status of 0 beside a non-zero error count is the shape that failure takes. The count is a grep and therefore a heuristic: a count above zero is worth reading, and a count of zero is not proof that a task did what it should.Verified on a NorSySS prod workspace pod on 2026-08-17, all six paths: no arguments exits 2; a directory holding no DESCRIPTION exits 2 and names the reason; a task that does not exist writes exit code 1; norsyss_data_delete writes exit code 0 after 106 plans in 1.4 minutes; a second call while the first ran exited 3 and surfaced through run_task_detached() as an R error; and no lock file remained afterwards.
.gitattributes pins *.sh to LF in the working copy. A checkout on Windows copied to a Linux machine otherwise carries CRLF, and bash reads the carriage return as part of the command. The failure reads $'\r': command not found, which does not name the cause.Task calls csutil::unnest_dfs_within_list_of_fully_named_lists(). It called splutil::, and splutil was in neither Imports nor Suggests. The two packages publish the same utilities under two names, and the NorSySS pods carry csutil alone. So upsert_at_end_of_each_plan = TRUE and insert_at_end_of_each_plan = TRUE both raised there is no package called 'splutil' on every pod, and nothing installed the package that would have fixed it.run_sequential() holds R/r6_Task.R:295 and :301. run_parallel_plans() holds :351 and :357. The parallel pair fails worse: the worker wraps its body in a tryCatch with five attempts and a five second sleep between them, so a missing package costs 25 seconds before it surfaces as Error in index N.DESCRIPTION declares csutil, pbmcapply and utils. pbmcapply::pbmclapply() at R/r6_Task.R:322 is how every parallel task runs, so a machine without it could not run one. utils ships with R, so it is always installed, but the dependency was still undeclared.tests/testthat/test-plan-end-write.R covers all four sites, one test each. Neither feature had a test, which is why the splutil calls survived. The two parallel tests mock pbmcapply::pbmclapply to run the worker in process, so nothing forks and the write is observable. Reverting csutil to splutil fails all four. The cs9 suite holds 387 passes, up from 368.Task$run() no longer calls future::plan(future::sequential) or foreach::registerDoSEQ(), and DESCRIPTION no longer declares future or foreach. cs9 never creates a future plan and never registers a parallel foreach backend, so both lines reset a state that cs9 does not set. The two %dopar% blocks that would have needed them are commented out, at R/r6_Task.R:425 and :523. data.table::setDTthreads() on the next line is a real reset and stays.26.8.18 declared future and foreach rather than removing the calls. That release is superseded on this point.Suggests declares devtools. devtools::load_all('.') at R/r6_SurveillanceSystem.R:414 and R/r6_TaskJob.R:72 is generated text, inside a glue::glue() string that cs9 writes to a temporary .R file. cs9 then runs that file in a child R process, so the feature needs devtools at run time. Process separation moves the dependency and does not remove it. Suggests is the right category, because one optional feature needs it and the package does not.CLAUDE.md for the sweep, and for the three matches it reports that are text rather than calls.DESCRIPTION declares foreach and future. Task$run() calls future::plan(future::sequential) and foreach::registerDoSEQ() at R/r6_Task.R:262 and :263, on the path every successful run takes. Neither package was in Imports, in Suggests or in NAMESPACE. A machine without them installed therefore raised there is no package called 'future' at the end of every task.26.8.17 added tests/testthat/test-fork-ordering.R, which drives the public Task$run() to completion, so CI ran the line for the first time and went red. The development pods carry both packages, so every local suite stayed green.tests/testthat/test-fork-ordering.R pins the order that Task$run() uses: it closes every database connection before it forks. Nothing asserted that order before. A refactor that moves the sweep reintroduces silent data corruption.pbmcapply::pbmclapply and drives the public Task$run(). The mock reads every connection at the fork boundary and returns without forking. A test on the private run_sequential() proves that the helper disconnects. It does not prove that the public path calls the helper before it forks.R/r6_Task.R go red. The first deletes the disconnect sweep and fails 2 assertions. The second moves run_parallel_plans() in front of the sequential setup and fails 3.test-fork-ordering.R asserts the length of both connection vectors before it asserts that both are open. all(logical(0)) is TRUE, so the open check would otherwise pass on an empty list.cs9 code changed in this release. The version moves because 26.8.16 is already published from a different tree.DBPartitionedTableExtended_v9 documents its one shared connection. The object owns it, every child borrows it, and only the parent's disconnect() closes it.$nrow() and $info() document the partition column that 26.8.16 added. It is character, it matches names(self$tables), and it is the last column, so table_name stays at position 1 and nrow at position 2.DBPartitionedTableExtended_v9$nrow(collapse = FALSE) now returns a partition column. The column holds the partition tag of the table on that row. $info() returns the same column. A caller no longer has to recover the tag from the table name.self$partitions_randomized and marks the matching row with keep := TRUE. The loop variable IS the tag at that moment. Both methods now write it into the row.partition is character, so it matches names(self$tables) and indexes self$tables[[...]] directly.partition is the LAST column. nrow(collapse = FALSE) returns table_name, nrow, partition, in that order. info(collapse = FALSE) appends partition after the columns that csdb returns, and moves nothing else. An existing caller that reads column 1 or column 2 by position therefore reads what it read before. table_name and nrow also keep their names, their types and their values. nrow(collapse = TRUE) still returns one number. info(collapse = TRUE) aggregates over every partition, so it has no single tag and returns no partition column.DBPartitionedTableExtended_v9$initialize() writes xxpxx for PostgreSQL and for SQLite, and PARTITION for the MSSQL fallback. norsyss.cs9 split the name on _PARTITION_ to find the partition with the fewest rows. Production is PostgreSQL, so the split returned NA, self$tables[[NA]] returned NULL, and the consultations import died on attempt to apply non-function. That task therefore never ran against PostgreSQL. Measured on a NorSySS pod on 2026-08-14.partition before the loop, beside keep, so the column is character from the start. The declaration is not what creates the column. A := whose i matches no row still creates it, and gives it the type of the assigned value. Measured with data.table 1.18.4 on 2026-08-14.tests/testthat/test-partition-safety.R pins the new column. It asserts set equality against names(pt$tables), because partitions_randomized shuffles the row order. It also asserts that pt$tables[[row$partition]]$table_name equals row$table_name for every row. That second assertion is the one that catches the original defect. No test names a separator, because the point of the change is that a caller does not know one.table_name is column 1 of nrow(collapse = FALSE) and that nrow is column 2, by position and by type. One asserts that info(collapse = TRUE) returns its four aggregate columns and no partition. One asserts that info(collapse = FALSE) appends partition last, against the base columns read from csdb itself.DESCRIPTION requires csdb (>= 2026.8.15). That is the release where DBConnection_v9 refuses a connection another process opened. Task$run_parallel_plans() forks with pbmcapply::pbmclapply and passes the table objects into the workers, and every partition of a partitioned table now shares one connection. A forked child that used the parent's connection received wrong results and no error.csdb 2026.8.14 carries the shared connection without that guard, so this floor names 2026.8.15 rather than the release it would otherwise pair with.cs9 source file changed in this release. The version moves so the raised floor reaches a distinct tree, because 26.8.14 is already published.Task$run() closes every connection in run_sequential() before it forks, so production never reached the corrupt state. Nothing asserts that ordering. csdb 2026.8.15 removes the dependency on it.DBPartitionedTableExtended_v9 now opens one database connection for the whole table, not one per partition. It builds a single csdb::DBConnection_v9 before the partition loop and passes it to every DBTableExtended_v9. A table with 106 partitions needed 106 simultaneous connections before this change. That exceeded the NorSySS PostgreSQL limit of 100 connections on 2026-08-13. The import died with FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute. The failure did not depend on the run size, because the partition count exceeds the budget before the first row is inserted.DBPartitionedTableExtended_v9$disconnect() now closes the shared connection once, at the parent. It looped over the partitions and closed each child before. Each child borrows the connection, so a child's disconnect() is a no-op in csdb 2026.8.14.DBTableExtended_v9$initialize() gains dbconnection as its eighth and last argument. It forwards that argument to csdb::DBTable_v9 in the eighth position. The argument defaults to NULL, which keeps the previous behaviour: the object builds and owns its own connection. SurveillanceSystem_v9$add_table() names every argument it passes and passes no dbconnection, so it is unaffected.DESCRIPTION requires csdb (>= 2026.8.14). That is the version where DBTable_v9$initialize() accepts dbconnection. Without this version floor, the incompatibility would fail at run time rather than at install time.DBPartitionedTableExtended_v9$drop_all_rows_and_then_upsert_data() now sends each row of newdata to the partition that its partition column names. It read self[[self$column_name_partition]] before. self is the R6 object and holds no field with that name, so [[ returned NULL. NULL == "a" returns logical(0), and a data.table indexed by logical(0) holds zero rows. The method therefore ran drop_all_rows() on every partition and then upserted nothing. The whole table lost every row, with no error and no warning. drop_all_rows_and_then_insert_data() reads newdata and is unchanged.DBPartitionedTableExtended_v9 now rejects a newdata that it cannot route, before any method destroys a row. This is the third defect of this shape in the class in one week, and the routing fix above does not cover it. private$check_for_correct_partitions_in_data() read newdata[[self$column_name_partition]], and [[ returns NULL for a column that newdata does not carry. unique(NULL) is NULL, !NULL %in% x is logical(0), and sum(logical(0)) is 0, so validation passed. Both drop_all_rows_and_then_* methods then computed NULL == i, which is logical(0). Each took zero rows for every partition, dropped all of them and wrote nothing back. The whole table lost every row, with no error and no warning. The check now rejects four things, in this order.
newdata that is not a data.frame. It names the class it received.newdata with no partition column. It names the column it expected.NA in the partition column.newdata that carries the partition column is still accepted. Clearing every partition is a legitimate operation, and tests/testthat/test-partition-safety.R pins it. Only a NULL, or a table without the column, is rejected.DBPartitionedTableExtended_v9 now writes a generated partition column with a valid := call, and it writes into a copy. Four methods held newdata[, .(self$column_name_partition) := part]. That form raises LHS of := must be a symbol, or an atomic vector. Every table with a non-NULL value_generator_partition therefore failed on its first write. No test built one, which is why the defect survived. The form is now newdata[, (self$column_name_partition) := part]. := also writes by reference. The old form therefore added a column to the caller's own data.table, as a side effect of a write. data.table::copy() now runs first. The copy is inside the value_generator_partition branch only, and not on the common path. A NorSySS production import is memory-bound against about 20 GiB of usable pod memory, so a copy of every newdata would double the peak.DBPartitionedTableExtended_v9 now handles a numeric partition value, which two separate defects broke. First, sample(x, length(x)) reads a length-1 numeric x as the range 1:x. A single partition numbered 9 therefore became a draw from 1 to 9. Both call sites, in insert_data() and in the partitions_randomized active binding, now index by a permutation: x[sample.int(length(x))]. That form has no such special case and is a no-op below length 2. Second, [[ on a list takes a numeric index as a position, so self$tables[[5L]] returned the fifth child rather than the child named "5". In the constructor it also grew the list to five elements, three of them NULL. All 16 self$tables[[...]] sites in R/r6_DBPartitionedTableExtended_v9.R now index by as.character(). A character partition value was never affected by either defect.DBPartitionedTableExtended_v9 now declares indexes as a public field, and initialize() assigns it. The class took indexes as a constructor argument and gave it to every child, but it held no field of that name. self$indexes was therefore NULL, and the drop_indexes = names(self$indexes) default in upsert_data() and in drop_all_rows_and_then_upsert_data() evaluated to NULL. Each child received an explicit NULL, which overrode the child's own names(child$indexes) default. Both defaults are now a plain NULL, so the runtime behaviour is identical and is now explicit. Nothing drops an index by default. Whether an index rebuild beats row-by-row index maintenance depends on four things: rows changed per child, index count and width, table size, and concurrency. A DROP INDEX also briefly blocks reads and writes. That needs measurement against real batch sizes. A caller who wants the drop now passes drop_indexes = names(pt$indexes).DBPartitionedTableExtended_v9$upsert_data() computed partitions_in_use twice from the same expression. One of the two lines is removed. Nothing observable changes. The method still writes to the partitions in the order newdata names them. It is not randomised, because no transaction spans the partition loop, so each child releases its locks before the next runs.DBPartitionedTableExtended_v9 now computes one route vector, and all four write methods route with it. private$check_for_correct_partitions_in_data() returns as.character() of the partition column. Each write method captures that vector and indexes with route == as.character(i), beside self$tables[[as.character(i)]]. One coercion now names the child and selects its rows. Validation used setdiff() on the raw values before, while the write methods used ==. Those were two independent answers to one question, and they agreed. An adversarial review said a factor column made them diverge and wiped every partition. Measured against the real class on 2026-08-14, it does not. setdiff() converts a factor to character, and == against a factor compares its labels, so both sides already read the same thing. The class is refactored so that the question has one answer, not because the two answers differed.DBPartitionedTableExtended_v9 now rejects a partition column that is not atomic. A list column passed the old check, because setdiff() also converts a list to character. setdiff(list("a"), c("a", "b", "c")) is therefore empty. drop_all_rows_and_then_upsert_data() then dropped every partition and died inside csdb on is.infinite(get(i)), which has no list method. A three-partition table holding six rows lost all six, and nothing failed until after the last drop. That is the fourth silent-data-loss defect of this shape in the class in one week. is.atomic() now rejects the column before the first drop. A factor is atomic, so a factor column still routes.DBPartitionedTableExtended_v9$initialize() now rejects a partition set that the class cannot use. It ran no check on table_name_partitions at all before. The check runs before the class opens its connection, so a rejected construction leaves no database connection open. It rejects four things.
NA.as.character(). self$tables[[""]] <- child matches no name, so it APPENDS. A set of c("a", "") built three children for two partitions.as.character(). self$tables[[name]] takes the first match, so a duplicated partition made one child unreachable while length(self$partitions) still counted it.names(self$tables) and all 16 self$tables[[...]] sites already used as.character(), so the character form was already the identity of a partition. One accepted input changes as a result. A newdata value of 1.0000000000000002 against a partition of 1 was rejected before, because setdiff() compares two doubles exactly. It now routes to the child named "1". Both values print as 1, and the partition column is stored as TEXT. A caller MUST NOT give one table two partition values that as.character() renders identically. initialize() now rejects that set.DBPartitionedTableExtended_v9$disconnect() can reopen the shared connection. csdb::DBConnection_v9$autoconnection calls connect() on every access, so any later use of that child opens a new connection. The child cannot close that connection again, because it does not own it. Only the parent closes it. Callers MUST NOT use a child after they disconnect the parent. A caller that does MUST disconnect the parent again. r6_Task.R calls disconnect() on the objects in the task's table list. A partitioned table is registered there as the parent, so the production teardown path is unaffected.DESCRIPTION no longer carries explicit Author and Maintainer fields. R derives both from Authors@R, which is the single source. The explicit fields named "Core Surveillance" as the copyright holder, while Authors@R names Folkehelseinstituttet. The 2026-08-06 sweep corrected Authors@R and left the free text behind, so the two disagreed on the legal entity.tests/testthat/test-shared-connection.R is new. It builds a three-partition table against a temporary SQLite file, and it asserts five things.
data.table::address().part column names.disconnect() on the parent closes the shared connection exactly once. The test counts the closes.disconnect() loop over every child leaves the partitioned table usable.disconnect() reopens the shared connection, and only the parent closes it again.tests/testthat/test-partition-routing.R is new. It builds a three-partition table against a temporary SQLite file, and it asserts three things. The file holds 35 assertions, and 29 of them fail when the routing defect is present.
drop_all_rows_and_then_upsert_data() sends every row to the partition that its part column names. The three partitions receive 2, 1 and 3 rows, so a row that reaches the wrong partition changes a count.drop_all_rows_and_then_upsert_data() and drop_all_rows_and_then_insert_data() write the same rows to the same partitions.drop_all_rows_and_then_upsert_data() empties a partition that newdata does not name.tests/testthat/test-partition-safety.R is new. It holds 21 tests and 178 assertions against a temporary SQLite file. The duplicated partitions_in_use line has no test, because its removal changes nothing that a test can observe. The whole cs9 suite now reports 298 passing assertions, up from 256.
drop_all_rows_and_then_upsert_data() with a newdata that lacks the partition column, and then re-reads every partition. Before the fix it reported no error and zero rows in all three partitions. An error alone would not prove that the rows survived.newdata that carries the partition column still clears every partition.value_generator_partition routes every row, and all four write methods leave the caller's data.table unchanged.partitions_randomized == 9. Under seed 5 the old sample() returned 2.c(5L, 7L) holds exactly two children, named "5" and "7", and a row routes to the child its value names.newdata that names exactly one numeric partition reaches that partition. This covers the second sample() call site, which is inside insert_data().pt$indexes holds the list the constructor received.check_for_correct_partitions_in_data() returns a character vector whose length is nrow(newdata). The test reaches private through pt$.__enclos_env__$private, and it is the only test that does.DESCRIPTION Authors@R now declares that holder with role = "cph".
It declared no copyright holder at all, and neither did any other package
in the fleet. Nothing in R CMD check reports that.CLAUDE.md now carries a Licensing section, so the year gets checked
rather than silently ageing.README.md, index.md and NEWS.md is now at most 25 words. 35 sentences were over that limit before, counted over the whole repository. They divide into 2 in roxygen, 15 in the vignettes, 16 in NEWS.md and 2 in index.md. The vignette count of 15 covers both the built .Rmd files and their .Rmd.orig sources. No claim, number, condition or attribution changed. The NorSySS figures, the 106-diseases and 378-locations counts, and the White & Valcarcel Salamanca attribution are unaltered.SET ROLE "" trap in vignette("backends"), the same trap in vignette("installation"), and the csdb version-floor entry in NEWS.md.CS9_PATH MUST NOT be empty. CS9_DBCONFIG_ACCESS MUST include config. The function named by data_selector_fn_name MUST return a named list.TaskJob method summary is a list. It was an indented block, which Rd collapsed into one paragraph..Rmd and its .Rmd.orig source. vignettes/_PRECOMPILER.R therefore still reproduces the .Rmd from the .orig.CS9_DBCONFIG_DRIVER=SQLite is accepted, matched case-insensitively. SQLite is a file rather than a server. CS9_DBCONFIG_SERVER and CS9_DBCONFIG_PORT therefore moved out of the always-required tier of check_environment_setup(). Only the server-based drivers now require them. A SQLite environment needs CS9_AUTO, CS9_PATH, CS9_DBCONFIG_ACCESS, CS9_DBCONFIG_DRIVER and one CS9_DBCONFIG_DB_<ACCESS> file path per access. It needs no CS9_DBCONFIG_USER, no CS9_DBCONFIG_PASSWORD and no CS9_DBCONFIG_SCHEMA_*.check_environment_setup() now rejects a CS9_DBCONFIG_ACCESS list that omits config. The four configuration tables are built from that access unconditionally, so its absence used to fail later and obscurely.reload_db_config() — new export, no arguments. Re-reads every CS9_DBCONFIG_* variable and rebuilds the configuration tables. A package that sets its own values in .onLoad() needs this. cs9 is a dependency, loads first, and reads the environment before that package runs. The reload is state-safe. It disconnects every table it replaces, so a repeated reload does not leak connections. It also empties config$tables before it rebuilds them. A reload against an invalid environment therefore leaves an empty table list, not tables describing the previous configuration.DESCRIPTION requires csdb (>= 2026.8.5) and carries Remotes: niphr/csdb. The bare csdb it had before let a resolver satisfy the dependency with any version. CRAN and RSPM serve version 2026.5.13, which predates csdb's SQLite backend. Installed against that, CS9_DBCONFIG_DRIVER=SQLite matched no branch in csdb and fell through to the generic ODBC arm. $connect() then failed with Can't open lib 'SQLite' : file not found, which names neither csdb nor a version. csdb 2026.8.5 is on GitHub and not on CRAN, so the floor alone would leave the dependency unsatisfiable. The Remotes field is what makes it obtainable. cs9 is not submitted to CRAN, so the field costs nothing.tests/testthat/test-sqlite-config.R now call skip_if_not_installed("csdb", "2026.8.5"). This is not redundant with the version floor. Nothing enforces an Imports version after installation. cs9 reaches csdb through csdb:: alone, so NAMESPACE holds no import directive. R therefore runs no version check at load time. Measured on 2026-08-05 against csdb 2026.5.13, R CMD INSTALL exits 0 and library(cs9) succeeds. The guard is keyed on the version and on nothing else, so it cannot hide a failure that is not a version mismatch.setup_database_tables() now builds all four tables into a local list and assigns config$tables once, at the end. Assigning each table directly meant a failure in the third constructor left a partially-populated config$tables behind, indistinguishable from a complete one.id is the database file path rather than [db].[schema]. A partitioned table's per-partition name uses the xxpxx separator that PostgreSQL uses. PARTITION is a keyword in SQLite's window-function grammar.cs9, writes six settings into .Renviron, validates them and then opens the database, all before the vignette mentions a server. The last step is deliberate. check_environment_setup() only checks the variables. It takes a $connect() on a configuration table to create the SQLite file and the config_log table in it. The PostgreSQL-in-Docker guide follows below.vignette("backends") — new. It puts the CS9_DBCONFIG_* environments for PostgreSQL and SQLite side by side. It has a table of what each backend does with every variable. It gives the tiers check_environment_setup() validates in, and the .onLoad() pattern that sets the variables from another package and then calls reload_db_config(). It carries no R-level detail: vignette("backends", package = "csdb") is the companion for that.CS9_PATH counts as a missing variable, so CS9_PATH= fails validation. And an unset CS9_DBCONFIG_ROLE_CREATE_TABLE reaches csdb as "" rather than NULL. "" is not the "x" no-role sentinel, so the PostgreSQL create_table can emit SET ROLE "". Both PostgreSQL blocks now set the variable explicitly..Renviron block in the installation vignette carried both of those traps, and had done so since before the SQLite work. It wrote CS9_PATH= with no value. The variable table below it called CS9_PATH "usually empty". CS9_DBCONFIG_ROLE_CREATE_TABLE was absent. A reader who copied the block got Missing required environment variables: CS9_PATH. All three are fixed.README.md names both backends and links to the two vignettes._pkgdown.yml indexes all five vignettes. cs9 and backends were missing from the articles: list; cs9 had been missing independently of this work, although the navbar links to it.DBI and RSQLite added to Suggests, for the new tests/testthat/test-sqlite-config.R.plnr as
the simpler option for a pilot or a resource-constrained setting.DESCRIPTION now declares Config/roxygen2/version in place of RoxygenNote, and every .Rd file was regenerated by that version. NAMESPACE is unchanged.TaskJob R6 class and run_task_sequentially_as_callr_bg_using_load_all() wrapper. A drop-in alternative to run_task_sequentially_as_rstudio_job_using_load_all() that works in editors without RStudio's job API (notably Positron, which does not implement runScriptJob). Spawns the task in a fresh callr::r_bg() process, so the current environment is not polluted. Captures output via a pipe. Streams the output back to the calling R console (prefixed with the task name) via later::later() polling. Includes $start(), $wait(), $is_alive(), $status(), $tail(), $kill().check_environment_setup()?cs9)check_environment_setup() documentation with detailed examplescopy_into_new_table_where now also copies indexes.copy_into_new_table_where uses tablock.upsert_at_end_of_each_plan and insert_at_end_of_each_plan can now take named lists as the return value from the action_fn.sc8::add_task_from_config_v8 the schema list is checked to make sure they are actually schemas. This will solve the issue where people incorrectly add non-existent schemas to the task.insert_data, upsert_data, drop_all_rows_and_then_insert_data are now the recommended ways of inserting dataaddin_load_productionload_folder_fn, which should dynamically check if a user has permission to write to a folder, solving permissions errorstm_get_schema_namesgranularity_time AND granularity_geo are now included in db censorssc8::config$plan_attempt_index now exists. When running plans in parallel, if a plan fails it is retried five times. This lets a user track which attempt they are on. This is mostly useful so that emails and smses are only sent when sc8::config$plan_attempt_index==1task_from_config_v8 and add_schema_v8db_insert_data, db_upsert_data, db_drop_all_rows_and_then_upsert_data are now the recommended ways of inserting dataupdate_config_datetime and get_config_datetime now automatically record database table updates as wellqsenc_save and qsenc_read to save/read to/from encrypted files.task_from_config_v3 sets a new direction for creation of tasks and management of tasksdescribe_tasks and describe_schemas help with automatic documentationtask_inline_v1 allows for easy inline task creationcopy_into_new_table_where allows for the creation of a new table from an old tabletask_from_config_v2write_data_infile now checks for Infinite/NaN values and sets them to NATask now includes action_before_fn and action_after_fnvalidator_field_contents_sykdomspulsen now allows baregion as a valid granularity_geotm_get_plans_argsets_as_dt provides an overview of the plans and argsets within a taskkeep_rows_where now also retains the PK constraints