Notes from Research Advisory Committee Meeting

MS Word version (37K)

Date: September 11, 2002, 10 a.m. - 3 p.m.

Location: University of California Office of the President, Room 9115, 1111 Franklin, Oakland, CA

Attendees: Ann Bishop (by phone), Christine Borgman (by phone), Lisa A. Bero, Colleen Cook, Michael Cooper, Cecily Johns, Gary Lawrence, Claire Le Donne, Laura Fosbender

Welcome and introductions; summary of progress

The project will likely extend three months past the original closing date, with no additional funding needed from the Andrew W. Mellon Foundation.

Cecily Johns, the Project Director, has several upcoming presentations about CMI.

CMI staff continue to gather surveys from patrons who have requested that experimental journals be recalled from storage. The volume of these requests is low.

User Preference Survey

Review of survey planning documents

The Inventory of User Preference Survey Questions categorizes and sorts questions by question type: preference, barrier, opinion, fact (demographics).

Other documents provided: Scope Statement, Final Report on Formative Interviews (previously distributed), and Systemwide Headcount by Campus.

Discussion

CMI staff would like assistance from the Research Advisory Committee on: adequacy of survey design, the wording of the questions and the question sequencing.

The clustering of questions by type is helpful. A desirable sequence of questions by type might be:

1. Usage questions
2. Preference questions
3. Opinion questions
4. Barrier questions
5. Demographic questions

Putting questions about usage first helps respondents to frame their answers about preferences, barriers, etc in the context of their experience. This will also reduce the respondents' cognitive load.

To further reduce cognitive load, CMI might also include a few easy-to-answer questions towards the top of the survey.

There seem to be too many preference questions; it may be better to reduce the number to 6-10.

The most salient reason that patrons like digital journals seems to be that they can access them anywhere, yet this question is not represented in a straightforward manner in the survey questions.

Connection speed seems to be an important factor that is missing from the questions. Generally, people can make the distinction between dialup service and everything else. Any other choices may be confusing to many respondents.

It might be helpful to anticipate how the data will be used in order to winnow the preference questions. Also, prior research has already shown a number of user preferences vis-à-vis digital format. For instance, we know that users do not like to read long items onscreen. Further, CMI may want to focus on issues over which UC has control, rather than on the quality of illustrations or other publisher-controlled qualities.

On the other hand, even preference information that is ostensibly outside UC control may be helpful for UC campus libraries to know, since it can guide the selection of journals for storage. Further, preference information may be helpful in persuading constituencies of the efficacy of future policy proposals.

The survey will likely be completed via different modes - telephone, web and print. Does this bias the results? CMI may want to analyze the results afterwards to see what effect the different modes of delivery had.

As far as sample size, the two survey contractors have estimated that anywhere from 8,500 through 13,000 responses may be needed. This will allow for a statistically significant breakdown by discipline and by campus. There was some question as to whether these breakdowns were needed. Still, CMI staff felt that gathering campus-specific data was important so that individual campuses might analyze their own data. While gathering data by narrow discipline might be difficult, a broader definition of disciplines, to include 4 categories (likely Arts & Humanities, Life & Health Sciences, Physical Sciences & Engineering, Social Sciences) will be achievable and useful, CMI staff believe.

Much of the survey execution will be in the hands of the contractor, such as locating survey participants, gathering enough responses, etc.

The inclusion of format (e.g., "I prefer digital journals because…") in the phrasing of preference questions could be misleading. A better way might be to focus on the characteristic exclusively, not the format.

The scenario-based questions that appear on John Vasi's report seem like they would be effective -- if answers were multiple choice. Open-ended questions on a survey will not get much response.

Another option would be to divide preference questions into two parts: importance and preferred format. On the other hand, this may be too complex for respondents. It may be better to simply ask about importance.

Respondents will not likely distinguish between Internet use and library-provided online resources. Efforts will be made to address this issue.

Journal Use Data

Review of data and analysis; consideration of counting methods for digital use

Data analysis indicates that the largest factor affecting journal use has been publisher. This may or may not be attributable to the various publishers' counting methods.

Further, CMI staff has been including abstract uses/views in article counts, wherever abstract counts are available. It would appear that abstract counts are inflating several publishers' statistics, and CMI staff is reconsidering whether abstract use should be conflated with article use.

Documents provided: Quarters 1-3 Detail Report, Preliminary Statistical Results from CMI Use Data: First and Second Quarters, Abstract and/or Article Usage Tabulation, Investigating Abstract/Article Display and Counting Methods.

Discussion

There are other factors besides counting method that might explain why different publishers might have different levels of journal usage. Presentation/usability of the material differs from publisher to publisher, as does the amount and efficacy of marketing. For instance, American Physical Society has made strenuous efforts to promote its digital offerings.

There may also be a bias for discipline; for instance, Physics faculty are more likely to consult digital resources than faculty in many of the Humanities.

Another issue is the length of the journal run. Study titles with more numerous back files may show higher use.

The counting of abstract views is problematic. CMI is running against most UC professional opinion. Further, it does now appear that abstract usage is skewing some publishers' statistics. For that reason, CMI staff will recalculate the statistics using only article views.

Review of special studies

Costs

CMI staff have collected Phase I costs from each campus. Salary information has also been collected. CMI will be using a generalized model created by Michael Cooper.

Staff will be identifying cost factors associated with ongoing storage and related operations. Michael Cooper will complete the model and associated analysis within the next 3-4 months.

Title characteristics

CMI staff have met with a group of UC librarians to identify and refine a group of journal characteristics to measure the 285 study titles against. It is hoped that measuring these characteristics will shed light on reasons for differences in electronic usage and help guide campuses in their (de) selection strategies for print.

Qualifying and quantifying graphics has been the most problematic issue.

The characteristic "Refereed" was initially included, then dropped from the study. It should probably be reinstated.

There are a number of issue-level items that could be added, such as Editorials, Editor's Notes, Editorial Board, Obituaries. This has the potential to be a quagmire.

Whether or not a journal is society-affiliated is an important characteristic that should be included.

ISI impact factors do not seem to be a reliable indicator of journal importance. A prior study shown that there is little correlation with use data. Paid circulation might be a better measure.

Will there be enough study uses to meaningfully correlate use data with the given characteristics? Unknown.

This seems like a lot of work for little return. It was decided that it would be better to wait for the survey results to come in. Then, if any characteristics have been identified by respondents, they might be examined in detail.

Campus characteristics

In anticipation that campus might be a factor in the usage data, CMI staff have compiled a list of campus metrics for which there is available data. The list is lengthy and needs winnowing.

Key metrics identified:

1. Undergraduate enrollment
2. Graduate enrollment (including professional enrollment)
3. Enrollment by discipline, distilled into four categories: Arts & Humanities, Life & Health Sciences, Physical Sciences & Engineering, Social Sciences
4. Amount of research funding
5. Academic staff

A helpful metric, if available, would be Research/Teaching Mix.

Next Steps

CMI staff will circulate the survey for comments once we have hammered out a final draft design. The survey contractors' proposals should reach staff within the week.

Comments on analysis of usage data are welcome anytime. Third quarter (April 1, 2002-June 30, 2002) electronic use data should be available in 6 weeks.

The next face-to-face meeting will most likely occur after the survey results are in.