Read these before upgrading: some numbers, names, labels and missing values are now different (and correct).
V7 37-43 (1) in SPSS, V7 37-43 .1 in SAS)
used to be read as whole numbers, so a value of 0000785 was 785 rather
than 78.5. They now get their decimal point, with a message listing each
column and its number of decimal places. With coerce_numeric = FALSE
the point is added to the text ("000078.5"). Values that already have a
decimal point are kept as they are, as in SPSS and SAS. Value labels and
missing values of these columns, which setup files write as numbers
(1.0, -9.00), are now matched by value. parse_setup()$setup has a
new decimals column.CLOTHING_V93, and PSID's repeated
"RELEASE NUMBER" becomes RELEASE_NUMBER_TA050001 for the 2005 wave.
Names are compared ignoring case, as SPSS and SAS do, so Age and AGE
count as the same name. Before, the data had duplicate names, so
data$CLOTHING only gave the first of them and dplyr and data.table
stopped with an error. A message lists the renamed columns.= inside a label no longer cuts the name short. "SEP FROM FATHER
< OR = 5" was named X5; it is now SEP_FROM_FATHER_under_OR_5.% in a label becomes _percent_ in the name, as < already became
_under_ (e.g. "1919 RT % VOTES CAST" is X1919_RT_percent_VOTES_CAST,
not X1919_RT_VOTES_CAST, which "1919 RT VOTES CAST" could also be).9, 09 or 0000009 matches any of them in the data.
Before, they had to be written exactly as in the data, so
IF V156=0000009 THEN V156=.; never matched the 9s in a 1-column field.
Columns the setup file says are text (SPSS (A), SAS $) still need
exactly the same text.0 now labels data written 0000000 (e.g. 7,997 "Not updated"
values in the 1960 UCR file that used to stay as 0000000). A column
that now has some labelled values is text rather than numbers: in the
1960 UCR file, columns such as JAN_LAST_UPDATE have "Not updated" where
they had 0. use_value_labels = FALSE keeps the codes.NA in text columns get their value labels:
e.g. 166,650 blanks in the 1960 UCR file are now "No return received" and
15,555 in the 2000 NIBRS batch header "Inactive", where they used to be
NA. (SPSS treats a blank text value as a value; SAS treats it as
missing but still gives it a format's label for a blank.) A blank or NA
text value without a label is still NA. In number columns a blank, NA
or a lone . is missing (a lone . used to be kept).(A) or an A
format, SAS $), as in SPSS and SAS, even when every value is a number,
so codes such as ZIP code 01001 keep their leading zeros. Before, such
a column became numeric with coerce_numeric = TRUE. In the SHR example
data, the rarely filled-in ages of victims 5 to 11 are now text like
those of victims 1 to 4, and empty text columns are character NA
rather than numeric NA. Use as.numeric() on a column you want as
numbers.LFM and LFF were ..._LOCAL_AUTHORITY and
..._LOCAL_AUTHORITY_LFF (one name, made unique) and are now
..._LOCAL_AUTHORITY_MALE and ..._LOCAL_AUTHORITY_FEMALE. 24 names in 6
of the 157 setup files used to check this release changed this way.ADD VALUE LABELS adds to them instead). Before, the first set
was kept. A message lists the columns whose earlier, different labels were
replaced.F7.1 don't imply decimal
places.%, =, full SAS labels). parse_setup(setup_file)$setup
lists every column's short name (column_number) and descriptive name
(column_name); short names haven't changed, and
use_clean_names = FALSE gives them.10^decimals, using its decimals in parse_setup(setup_file)$setup.use_value_labels = FALSE keeps the codes, and as.numeric() turns a
text column of numbers into numbers.missing_values = "all", including ones that are commented out in the setup file).read_ascii_setup(), spss_ascii_reader() and sas_ascii_reader() have
a missing_values argument. Many setup files have their missing values
section commented out (* MISSING VALUES in SPSS, inside /* */ in SAS)
so the codes stay in the data unless you choose otherwise:
"all" (the default, as before) uses every missing value in the file,
"active" only those that aren't commented out, and "none" none of
them. When commented-out missing values are used, a message says where
they are and how many values they made NA. parse_setup()$missing has a
new commented column.-9 in a (2) column is -0.09, not the
code -9 'Blank'). As in SPSS and SAS the code isn't used, but it suggests
the data doesn't match the setup file's decimal places."asciiSetupReader_message" (and one for each
kind), so they can be silenced with suppressMessages(). Missing values
that also have a value label keep the label (as before); the
documentation now says so.V1 V2 1 'Yes', V1 TO V5 (9)) and starting on
the command's own line, ADD VALUE LABELS, several MISSING VALUES
sections in one file, commented-out VALUE LABELS between other sections,
variables named VALUE or LABELS, and SAS formatted input
(@1 V1 $CHAR2. @3 V2 4.1).V1 () clears them, ALL means
every column, / can separate columns on one line, THRU can be in any
case, and long or decimal ranges (1000000 THRU 9999999,
99.5 THRU 99.9) are compared as numbers instead of being listed one
value at a time.>=/GE alone (98 thru highest), <=/LE,
GE a AND LE b, EQ, lower-case names and statements over several
lines. Statements that aren't "IF v = ... THEN v = .;" (e.g. a recode to
another value, or a condition on another column) used to be misread as
missing values; they're now skipped, with a message.IF v = 1 THEN v = 5;, SPSS RECODE v (1=5)) aren't applied; a message
lists them. Only a setup file's layout, names, labels and missing values
are used, as before, but such statements used to be misread.SKIP (SPSS DATA LIST), /FIRSTCASE (GET DATA) and FIRSTOBS (SAS
INFILE) skip header lines in the data.@ pointers (INPUT V1 2. V2 $3.;), INPUT on
the same line as the first column, blank lines inside INPUT and
commented-out INPUT statements are handled."asciiSetupReader_error": a file for delimited data
(GET DATA /ARRANGEMENT = DELIMITED, DATA LIST FREE or LIST, or SAS
DLM=/DSD, GitHub #13), a
file of value labels without column positions (such as PSID's separate
formats files), a file without a DATA LIST or INPUT section, or a layout
the package can't read: several records per case (RECORDS=, /2,
SAS #2), columns given by format only (ID (F3) NAME (A5)) or SAS variable lists in
parentheses. These used to fail with "invalid 'row.names' length" or
"argument of length 0", or read the wrong columns.FORMAT V1 V2 YNF.;), numbered and positional lists
(V1-V5, Q08-Q10, A1_1-A1_3, V1--V5), formats without value labels
(8.2, DATE9.) and ATTRIB ... FORMAT= (also after LABEL='...'). Format names match in any case, and a format defined
twice uses the later definition, as in SAS.VAR LABELS), several
columns on one line, / between columns, the first column on the
command's own line and labels joined with +; in SAS LABEL, several
columns on one line, blank lines, labels over several lines, a ; inside
a label, LABEL after another statement on its line, and
ATTRIB V1 V2 LABEL = '...' (for every column listed) whatever else
ATTRIB sets. A LABEL statement inside a /* */ comment isn't used.VALUE F 1, 2, 3 = 'Low';) give each code the label.
Value ranges (1 - 97 = 'Actual hours') and OTHER aren't applied, so
the real numbers are kept; a message lists them.AA 1-4(A) BB 5-8(A)), formats and
decimals anywhere on a line (V1 1-5 (F,2) V2 6-7), columns on the DATA
LIST line itself, a name and its position on different lines, /* */
comments, names with @, #, $ or ., and a missing closing period.
As in SPSS, names listed before one range share it equally (A B C 1-6,
Q1 TO Q3 3-5). GET DATA formats such as AUTO and ADATE10 and
/VARIABLES= without spaces are read, and a second record given by a
bare / stops with the several-records error. All 157 setup files used
to check this release are read exactly as before.inst/testdata to tests/testthat/testdata, so they're no longer
installed with the package (about 4 MB less).select_columns: a column number past the last column stops with an
error instead of crashing R (negative numbers still drop columns); a
column asked for twice is read once, with its implied decimals; a name
that isn't found is listed, with a note on which names changed in this
version; and selecting a name that another column also had says what
that column is called now.97 THRU HI, LO THRU -1, SAS IF V1 >= 97 THEN V1 = .;) no longer
stops read_ascii_setup() with "'names' attribute [1] must be the same
length as the vector [0]".INPUT V1 1-2 V2 $ 3-4 V3 5-8 .2;) is read token by
token, as formatted input is, so a statement or comment after the ; on
INPUT's line (INPUT V1 1-2 V2 3-4; RUN;), a /* */ comment inside
INPUT and single-spaced columns (SEX $ 8 INCOME 9-15) no longer drop or
misplace columns.97 THRU 99.
parse_setup()$missing still lists a range of whole numbers value by
value."anti-jehovah's witness" became
"anti-jehovah s witness" and 'Don''t know' became "Dont know"). This
affected both SPSS and SAS setup files.'Wife/"Wife"') is
kept, and so are = and ;, which used to become a space or be dropped.2 'Wife' 3 'Common-law husband' gave a code
"2 Wife 3"), a code written '''', quoted words inside a label
('farmer, ''profits''', 'Twelfth grade, "High School", GED'), and
SAS labels split over several quoted pieces, whose end became a separate
code.+ (at the end of a line or the start
of the next) are joined into one label, and a double-quoted code followed
by more pairs on the same line ("02" "two" 3 'three') no longer runs
into the next pair.coerce_numeric = TRUE now checks every row before converting a column to
numeric. Previously files over 50,000 rows only had a random sample of rows
checked (and a file of exactly 50,000 rows only 5,000), so a column with
rare text values could be converted to numeric, silently turning those
values into NA, differently on every run. The sampling also changed the
user's random number stream.make_sps_setup() works again on R >= 4.5.0 (it failed with "NA in
coercion to boolean"), works when col_labels isn't given, and no longer
leaves a sink() open if it fails. A file_name ending in ".SPS" no
longer gets a second extension, and one merely ending in "sps" (e.g.
"datasps") is no longer cut short.labels
must be unique" when the value X itself appears in the data.make_sps_setup()'s missing_values accepts column names as well as
labels. Previously a column name was treated as a missing value of the
column before it.DATA LIST can now be read.spss_ascii_reader() and sas_ascii_reader() now call
read_ascii_setup(); results and arguments are unchanged.vroom not handling embedded nulls.readr to vroom to
significantly increase read time.read_ascii_setup() function interactively. This allows
users to select every option in the popup window.parse_setup to be in same order
as in setup file.read_ascii_setup) that will read in data for both
.dat+.sps and .dat+.sas pairs. This is just a helper function that calls
spss_ascii_reader or sas_ascii_reader.