Reliability of OSCAR Occupancy, Census and Staff Data: A Comparison with the Ohio Department of Health Annual Survey of Long-Term Care Facilities Jane K. Straker Scripps Gerontology Center Miami University Oxford, OH 45056 Technical Report Series 3-01 December 1999 SGC0074 BACKGROUND AND SIGNIFICANCE In recent years, policymakers, planners, and consumers have recognized the importance of available, accurate information about health care facilities and services. Consumers use data comparisons to assist in choosing providers; planners use data to project service needs and initiate new services, and policymakers need reliable data to project program costs and budgetary needs. Some data are gathered through compliance and monitoring activities, other data sources, such as the National Long-Term Care Survey, are specifically gathered for research purposes. Because compliance and monitoring data include a population of facilities rather than a sample, they provide an important source of information about specific facilities and programs. Unfortunately, there are often problems when data that are collected for one function, such as compliance, are used for research purposes. This paper illustrates some of the problems involved in using data collected from certification of long-term care facilities for research and consumer information purposes. Every year, each nursing facility in the United States undergoes a survey and inspection process. For facilities that are eligible for reimbursement under federal Medicare and Medicaid programs, data are collected by each state to provide information for the federal certification process. These data include information about the conditions of residents, the number of residents receiving payments from different sources in the previous month, the hours worked by staff in different positions, the availability of special units, and services provided in the facility. These data are collected and maintained by the Health Care Financing Administration (HCFA). The database, commonly referred to as OSCAR (Online Survey, Certification, and Reporting) is the largest single source of information about nursing facilities in the United States. However, because not every nursing facility is certified for Medicare or Medicaid, these data do not describe the entire population of U.S. nursing facilities. In addition, because the data are collected as part of the ongoing yearly survey process, at any one time the database includes data that are potentially eleven months out-of-date for some facilities while being quite current for others. 2 However, given that the OSCAR database is the only nationwide source of data, it has important potential uses, particularly when combined with information from the Nursing Facility Minimum Data Set (MDS), a database of information about the physical conditions and cognitive and behavioral problems of nursing home residents. As of April 1, 1999 these data are downloaded from individual states to a central HCFA repository. Prior to this time, individual states collected and maintained their own MDS data. To complement the information collected by HCFA, many states conduct their own data collection efforts about all facilities in their state. Since not all facilities are certified for Medicare or Medicaid, OSCAR cannot provide a complete look at nursing homes in individual states. The Ohio Department of Health (ODH) collects data about the majority of licensed nursing facilities through their Annual Survey of Long-Term Care Facilities. This survey is a self-report that is administered by mail, and facilities licensed by the Department of Health are mandated by law to complete it. However, because of different licensing entities skilled nursing homes in hospitals, county homes, and facilities in Hamilton County (Cincinnati) do not have to complete the ODH survey in order to maintain their licensure. Extensive follow-ups are used to ensure 100% completion of all other nursing facilities (Ohio Dept. of Health, 1997). Each facility reports information about services offered, number and type of beds (certified and non- certified and beds licensed as nursing home or residential care facility) and available bed days, number and type of staff, the census of residents with different payment sources, average daily charges, and admission and discharge information. The surveys are generally mailed in February and ask facilities to report information about a particular week in October of the previous year. Thus, in March 1999 facilities were providing information about October 1998. The advantage of this strategy is that all facilities are reporting information about the same time period; the disadvantage is that by the time 1998 data are collected, entered and analyzed, there is at least a one-year time lag. That is, data from October 1998 may not be ready for use by October 1999. Data from the survey are edited and compared with previous surveys. For example, a facility that reported 80 beds last year and 800 this year will be checked against licensure records and the 800 will be corrected to 80. Over time, researchers have noted discrepancies between OSCAR data and other data sources. For example, the 1995 Nursing Home Statistical Yearbook (Cowles, 1996) based on OSCAR data reports that Ohio has 86,134 certified nursing home beds. The Ohio Department of Health Survey for 1995 reported that there were 88,017 certified nursing home beds (excluding Intermediate Care Facilities for the Mentally Retarded). While some discrepancies might be explained by differences in the times data were collected, a difference of over 2,000 certified beds is likely due to some other reason. In addition, OSCAR reports the existence of an additional 20,750 non-certified beds within certified facilities for a bed 106,884 nursing home beds in certified facilities. The ODH Survey reports 102,998 total nursing home beds in all certified and non-certified facilities. Interestingly, the ODH numbers show 4,000 fewer beds, while including data from 57 non-certified facilities not included in OSCAR. Although these differences have 3 frequently been observed there have been few attempts at rigorously attempting to explain these, and other variations between OSCAR data and other data sources. This study compares 1995 data from OSCAR to 1995 and 1997 survey data from the Ohio Department of Health to estimate OSCAR's reliability, to determine where discrepancies exist, and to explain the reasons for those discrepancies. The goal of the study is to provide guidelines for adjustments or selections that might be made to OSCAR data in order to improve their reliability. The following research questions will be addressed: 1) What factors are responsible for the differences between the two data sources in relatively straightforward areas such as the number of certified beds? 2) Do these discrepancies carry over to other areas such as staffing and resident census? 3) If researchers wanted to use OSCAR data in combination with MDS data, what kinds of adjustments could be made to OSCAR to improve its reliability? METHODS Using a file of licensing information from the Ohio Department of Health, facilities were matched based on their Ohio license number and Medicare/Medicaid provider number. For 1995, information on 900 facilities had matches from both data sources. Although the ODH data included 1279 nursing facilities, some of these were not certified and would not be expected to have OSCAR data; others were Intermediate Care Facilities for the Mentally Retarded, which also do not have OSCAR data. Other facilities may have closed, changed licenses, or opened during the year. There were 43 facilities in the OSCAR database that could not be matched with ODH data and 379 facilities in the ODH database that had no comparable data in OSCAR. Comparisons were made in the following areas: 1) Number of beds, both certified and in total; 2) Number of residents; 3) Number type of nursing staff—RNs, LPNs and CNAs. Descriptive statistics provide comparative information while Paired-Sample Correlations and T-tests indicate the extent and significance of variation between the two data resources. FINDINGS Total Beds 4 The first area of comparison was an attempt to understand the discrepancies between the number of beds reported in OSCAR and the numbers reported by ODH. When comparing the same facilities some minor discrepancies are expected since facilities might add additional nursing home beds, reduce the number of beds by converting some to assisted living, or other actions that would result in slightly different overall numbers between the two data sources due to the different times data were collected. The ODH Survey asks respondents to report the total number of nursing home beds in their facilities on Jan.1 and Dec. 31. Although a moratorium on beds has been in effect in Ohio since 1993, new beds that had been approved prior to the moratorium were still being added. Facilities reported a total of 1200 additional beds in these facilities that were not available on January 1. The total as of December 31 is still lower than that reported by OSCAR by about 6500 beds resulting in a correlation of .65 between the two data sources. Given such a tangible item as the number of beds and the impact of this number on determining staffing levels, occupancy rates, etc. an additional explanation for the discrepancy was sought. Certified Beds In contrast to the overall discrepancies discussed above, the reliability of data on certified beds between these two data sources is quite high (see Table 1). The minor discrepancies shown could be accounted for by a number of nursing facilities making changes in the actual number of nursing home beds available during the year. For example, changing a semi-private room to a private room, closing a wing for remodeling when occupancy was low, or converting nursing home beds to residential care facility beds which are not certified by Medicare or Medicaid would all change the number of certified beds. However, when using data about Medicaid or Medicare certified beds in Ohio, there is a high degree of agreement between these two data sources; correlations ranged from .88 for dually certified beds to .99 for Medicare certified beds. Beds in Facilities with Residential Care The discrepancy in the overall bed numbers might be explained by how the information about the total number of beds is collected. Differences could be occurring in the way facilities report their total number of beds in the ODH survey, or in the way that surveyors report information about the total number of beds in OSCAR. For example, many nursing homes in Ohio have combined residential care facilities and nursing homes. Residential care is primarily custodial and is not certified for Medicare or Medicaid. However, some surveyors may include these beds when compiling information for OSCAR. In order to determine if this was occurring, we compared facilities having some residential care beds with facilities having only nursing home beds. As shown in Table 1, if comparisons are limited to only those facilities having both nursing home and rest home beds, ODH data show about 10 more beds per facility. For those facilities that have no residential care beds (see Table 2), the average number of beds is about 10 higher in OSCAR data than in ODH data, and the total discrepancy increases from 6500 to about 7500 total beds. The average number and paired-sample correlations of certified bed measures shows little change from all nursing homes to this 5 subgroup that includes no residential care facility beds. Clearly, differences in reporting on residential care facility beds do not explain all of the discrepancy in the total number of beds. Beds in Hospital-Based Units An additional analysis compared our matched data from ODH 1995 and 1997 to determine whether the 1995 ODH data on the total number of beds seemed reliable. Because skilled nursing units in hospitals are not required to complete the ODH survey there is a wide swing in the number of facilities reporting from year to year; we needed to insure that the discrepancies were not due to missing data on these units in the 1995 ODH data. In 1995 there was data in both OSCAR and ODH for 44 Hospital-Based Units; 49 units had matching data in 1997. We examined the 1995 data on these individual facilities in both OSCAR and ODH. As shown in Table 1 we noted a huge discrepancy between OSCAR data, with OSCAR reporting 9,753 beds in 44 units, compared to 1,832 beds reported by ODH. The number of residents also varies, although not so dramatically. ODH data report 1501 residents in these units; OSCAR reports 1405. A case-by-case comparison of the data reveals that for most hospital-based units OSCAR data reflect the entire number of hospital beds, rather than only those included in the skilled nursing unit. Eight out of 44 hospital-based units, however, report the same number of beds in OSCAR as they report for nursing facility beds in ODH. No doubt, this also explains why OSCAR data show an average occupancy rate of 24% for these units, compared to 80% shown by ODH. In summary, the number of certified beds in OSCAR shows a high correlation with the total number of beds in ODH. For hospital based units, the number of certified beds from OSCAR data is clearly the more accurate measure since many of these units are also totally certified. Using the number of certified beds from OSCAR to compute occupancy rates for hospital based units would provide more reliable occupancy findings than calculations based on the total number of beds. In order to determine whether these assumptions about the reasons for differences are accurate, an individual from the Ohio Department of Health, Bureau of Healthcare Standards and Quality was interviewed. She agreed that surveyors are requested to report the total number of beds in the facility. For hospital based units, this includes all the beds in the hospital. For nursing homes with residential care beds, this includes both types of beds. She agreed that in actuality, OSCAR data are not collected in such a way as to allow for accurate reporting and counting of total licensed nursing home beds. Resident Census As shown in Table 4, ODH data report a resident census over 5,000 residents higher than reported in OSCAR. Although the correlation is fairly high, this overall difference in numbers will likely result in dramatic differences in overall occupancy rates as well. (Occupancy in each facility is computed as the number of beds divided by the number of residents, multiplied by 100). As shown, the correlation between occupancy rates is low (.48), and there is an average difference of about 6% between ODH and 6 OSCAR occupancy rates. Part of this is due to the discrepancy in the total number of beds noted earlier; part is due to the discrepancy in numbers of residents. Regardless of the sub-group of facilities examined the ODH data consistently report more residents than OSCAR data. For example, even among facilities without any rest home beds ODH data show more residents than OSCAR data. These discrepancies have important considerations for the examination of staff/resident ratios. Staff/Resident Ratios The last area examined is that of staffing levels in nursing homes, with emphasis on nursing staff levels. Table 5 shows a comparison of the average number of nursing hours per week filled by nurse aides and orderlies, licensed practical/vocational nurses, and registered nurses. Unlike the data on the total number of residents in a facility, OSCAR data show higher numbers of staff hours than those reported by ODH. There are several possible reasons for this discrepancy. First, the OSCAR data are collected by asking for the number of hours worked in the last pay period by full-time, part-time, and contract staff. If the last pay period is longer than fourteen days, then the number of hours worked in the previous fourteen days is reported. Within one facility, administrators paid monthly would report the hours they worked in the last fourteen days, while nurse aides might be paid weekly and their work hours would reflect only a 7-day pay period, not 14 days. Again, the Ohio Department of Health, Bureau of Healthcare Standards and Quality verified this assumption. They verified that because of the way these data are reported it is impossible to accurately determine the number of hours worked by each job classification per day. It is not clear whether the total number of hours should be divided by 7 or 14 (or some other pay period length) in order to determine daily hours and the number of full-time equivalent staff. ODH data are reported by collecting the number of full-time staff, and the total number of part-time hours in each job classification. Comparisons were made between ODH and OSCAR data to examine the extent of this problem. The total number of hours in three job classifications were divided by 14 (the number of days data were reported for) to arrive at daily totals for CNAs, LPNs, and RNs for OSCAR data. For ODH data the number of full-time staff was multiplied by 36 (hours specified by ODH) and then divided by 7 to arrive at an average daily number of hours worked. As shown, correlations between the two data sources for the average number of hours worked per day ranged from .56 to .73. These are not as poor as expected given the 7 or 14 day reporting problem mentioned in OSCAR. OSCAR also separates out contract staff into a separate category while ODH asks their hours to be included in full-time staff and part-time hours categories. It is possible that facilities are not including them despite this instruction which could account for the lower averages drawn from ODH data. As an example, contract hours make up about 1.3% of the total aide hours reported in OSCAR data. Increasing the ODH data by 1.3% resolves the discrepancy between the data sources somewhat. However, given the weak correlations between the two data sources, our test of reliability fails to verify either of these two data sources as reliable in the area of staffing levels, and staffing hours per resident. 7 IMPLICATIONS There is wide variation on the extent to which ODH and OSCAR data reliably report information about Ohio Nursing Homes. Some areas, such as the number of certified beds are highly reliable, while others, such as the number of residents, or the number of staff hours worked show very low agreement between the two data sources. This report has suggested some reasons for these discrepancies. In order to improve reliability of OSCAR data for examining facility characteristics such as staff ratios and occupancy rates the following changes in the data collection process should be made. 1) Discontinue the practice of recording all beds in the facility in the bed totals in OSCAR data. Among hospital based units and facilities with residential care beds this results in over counting of beds, and erroneously low occupancy rates. This is particularly important given HCFA's recent posting of facility occupancy rates on their Nursing Home Compare web site. They suggest that high occupancy indicates high demand or a waiting list, while questions should be asked about homes with a very low occupancy rate. When occupancy rates are an artifact of miscounting of beds, some facilities are at a disadvantage through no fault of their own. In addition, when occupancy levels may be used to show changes in utilization of nursing homes over time, accurate data is important for planning and policy-making. 2) Count staff hours worked during a clearly defined period, such as the week prior to the survey. Accurate reporting of staff hours would allow for examination of staff ratios in comparison to quality indicators, nursing home resident characteristics, and other facility characteristics. Given the current national interest in staff shortages in long-term care, accurate counting of existing staff, utilization of contract staff, and the relationship of staffing levels to quality indicators and survey deficiencies are important policy questions. These cannot be pursued without adequate, accurate data. 3) For current purposes, examine facilities that are 100% certified, without residential care beds, and not hospital-based. The most reliable data were found among certified beds and certified residents. A sample from these facilities would provide ample variation to examine staffing patterns, quality indicators, and other important relationships between resident and facility characteristics. 8 REFERENCES Cowles, C.M. (1995). Nursing Home Statistical Yearbook, 1995. Tacoma, WA: Cowles Research Group, Inc. Ohio Department of Health (1997). 1996 Annual Survey of Long Term Care Facilities. Columbus, OH: Author. 9 Table 1 Mean Number of Ohio Nursing Home Beds in 1995, Ohio Dept. of Health and OSCAR Data ODH Data OSCAR r Jan. 1, 1995 Licensed Capacity (sd) Dec. 31, 1995 Licensed Capacity (sd) Total Beds 1/1/95 Total Beds 12/31/95 Medicare Certified Beds in all Facilities (sd) (n=900) Medicare Certified Beds in Facilities with Medicare or Dual Beds (sd) (n=644) Dually Certified Beds in All Facilities (sd) Medicaid Beds in All Facilities (sd) Total Certified Beds (sd) Beds in Hospital-Based Units (sd) Total Beds in Hospital-Based Units (n=44) Beds in Nursing Homes with Rest Home Beds (sd) Total Beds in Facilities with Rest Homes (n=90) Total Certified Beds 102.5 (62.63) 103.9 (63.05) 92,267 93,534 2.05 (8.23) 2.87 (9.62) 35.89 (48.62) 54.32 (52.30) 92.26 (56.91) 41.64 (58.06) 1,832 143.41 (69.27) 12,907 83,037 111.23 (73.95) 111.23 (73.95) 100,106 2.02 (8.12) 2.83 (9.49) 36.14 (48.61) 53.49 (51.97) 93.38 (57.00) 221.66 (174.93) 9,753 132.80 (65.06) 11,952 84,045 .648*** .647*** .999*** .900*** .883*** .995*** .952*** .070 .844*** ***p < .001 10 Table 2 Mean Number of Beds in Ohio Nursing Homes With No Licensed Residential Care Facility Beds in 1995, Ohio Dept. of Health and OSCAR Data ODH Data OSCAR r Jan. 1, 1995 Licensed Capacity (sd) Dec. 31, 1995 Licensed Capacity (sd) Total Beds 1/1/95 Total Beds 12/31/95 Medicare Certified Beds (sd) Dually Certified Beds (sd) Medicaid Certified Beds (sd) Total Certified Beds (sd) Total Certified Beds 98.47 (60.47) 99.54 (60.80) 79,762 80,627 1.94 (7.97) 35.76 (49.36) 55.39 (52.19) 93.08 (57.38) 75,395 108.83 (74.52) 88,154 1.93 (7.93) 35.70 (49.02) 54.53 (51.88) 93.57 (57.85) 75,788 .62*** .62*** 1.00*** .884*** .994*** .966*** ***p .001 N=810 11 Table 3 Mean Number of Beds in Ohio Nursing Homes Excluding Hospital-Based Units in 1995, Ohio Dept. of Health and OSCAR Data ODH Data OSCAR r Jan. 1, 1995 Licensed Capacity (sd) Dec. 31, 1995 Licensed Capacity (sd) Total Beds 1/1/95 Total Beds 12/31/95 Medicare Certified Beds (sd) Dually Certified Beds (sd) Medicaid Certified Beds (sd) Total Certified Beds (sd) Total Certified Beds 105.74 (61.14) 107.13 (61.65) 90,511 91,702 1.41 (7.39) 36.57 (49.36) 56.29 (52.06) 93.08 (57.38) 81,236 105.55 (59.59) 105.55 (59.59) 90,353 1.38 (7.25) 36.76 (49.02) 54.53 (51.88) 93.57 (57.85) 82,216 .96*** .96*** 1.00*** .87*** 1.00*** .966*** ***p , .001 N=856 NOTE: OSCAR Data are from the date of survey, not a particular date. Only facilities having both OSCAR and ODH data are included. 12 Table 4 Mean Number of Residents in Ohio Nursing Homes in 1995, Ohio Dept. of Health and OSCAR Data ODH Data OSCAR r Residents per Facility Total Residents (sd) Residents per Facility in Facilities with no Rest Home Beds (sd) Total Residents in Facilities with no Rest Home Beds Residents in Hospital-Based Units (sd) Total Residents in Hospital-Based Units Medicare Residents (sd) Total Medicare Residents Medicaid Residents (sd) Total Medicaid Residents Occupancy Rate (%) (sd) Occupancy Rate in Non-Hospital and Non-Rest Home Facilities (%) (sd) Occupancy Rate in Hospital-Based Units (%) (sd) 92.55 (55.32) 82,554 90.13 (54.84) 72,284 34.11 (47.49) 1,501 5.90 (7.04) 5,314 62.10 (40.80) 47,569 88.89 (12.11) 90.48 (10.25) 80.36 (18.79) 86.24 (54.38) 77,270 86.44 (55.06) 69,670 31.93 (53.78) 1,405 5.81 (8.73) 5,232 62.88 (41.28) 48,167 82.16 (24.69) 87.33 (18.02) 24.50 (28.09) .909*** .950*** .982*** .638*** .941*** .473*** .481*** .071*** ***p .001 NOTE: OSCAR Data are from the date of survey, not a particular date. Only facilities having both OSCAR and ODH data are included. 13 14 Table 5 Mean Hours of Nursing Care in Ohio Nursing Homes in 1995, Ohio Dept. of Health and OSCAR Data ODH Data OSCAR r Nurse Aide Hours Per Day (sd) LPN Hours Per Day (sd) RN Hours Per Day (sd) Total Nursing Hours Per Day (sd) Nurse Aide Hours Per Resident Per Day (sd) LPN Hours Per Resident Per Day (sd) RN Hours Per Resident Per Day (sd) Total Nursing Hours Per Resident Per Day (sd) 182.39 (123.30) 66.10 (50.77) 43.95 (36.26) 292.45 (191.94) 1.99 (.68) .75 (.44) .55 (.63) 3.30 (1.29) 194.09 (204.87) 69.08 (99.38) 36.48 (38.53) 299.64 (325.03) 2.51 (2.98) .97 (1.47) .67 (1.71) 4.14 (5.53) .585*** .561*** .731*** .613*** .083* .310*** .344*** .223*** ***p ≤ .001 Note: Cases reporting 0 hours were excluded from analysis.