The aim of this study was to develop a useful method for evaluating patient information leaflets by comparing the result of expert and patient evaluations. The hypothesis was that if there was a correlation between the results of these evaluations it would be acceptable to only do expert review for patient information leaflets currently in use.
Experts and patients examined patient information leaflets from 30 common medicines, following European Union Directive 92/27 EEC, which outlined the legal requirements for patient information leaflets. The scoring system allowed a direct comparison between the outcomes of expert and patient examinations. We found a significant correlation between the expert and patient examinations of the content of the patient information leaflets. We conclude that patient information leaflets that score above average on the expert examination of content (ie, adherence to the directive) will also score above the average on the patient examination.
Key Words Patient information leaflets; Readability; Comprehensibility
INTRODUCTION
It has been mandatory for medicines sold within the European Union (EU) to include patient information leaflets with each medicine pack since 1999. The requirements for the leaflets are specified in Directive 92/27 EEC (1).
Patients' understanding of the purpose, benefits, and risks of medications is often a prerequisite for successful therapy. Health professionals play an important role in this information process, and a desirable situation is described as concordance between the health care professional and the patient (2). It is an advantage for patients to obtain both oral and written information (3) and the patient information leaflets are intended to provide the appropriate information in written form. However, patients' ability to process printed medical information depends upon a complicated interaction between the patient and the printed material, as recently summarized by Morris and Aikin (4). This may lead to great variability in the use of the patient information leaflets. Raynor and Knapp (5) have shown that nearly 20% of the patients failed to notice the package inserts, and among those who could recall that they had received a leaflet, only 60% had read part or all of the text. Raynor and Britten (6) also argued that patient information leaflets could undermine the ability to achieve concordance.
If printed medical information is to be of benefit it must be read, understood, and acted upon. Attempts have been made to estimate the impact or outcome of printed patient information (5,7), but the effects observed thus far are small. One contributing factor to the lack of success could be that the leaflets need to be improved. It has been stated that the leaflets not only should be scientifically correct but that they also should be tested for comprehensibility and relevance (8).
Recent studies of patient information leaflets in the United States have shown significant differences in the reading level of leaflets from different pharmaceutical companies (9). The European Agency for the Evaluation of Medicinal Products (EMEA) has recently issued a recommendation (10) to companies to conduct user tests of their patient information leaflets. Such tests would require the involvement of patients (8,9) and would, therefore, be complicated to conduct and difficult to standardize. Furthermore, tests of all patient information leaflets in circulation in a country at any given time would be a very large endeavor. In this study we have, therefore, tried to devise a method for the efficient evaluation of patient information leaflets.
AIM OF THE STUDY
The aim of this study was to develop a simple and reliable method that would allow tests of the readability and comprehensibility of patient information leaflets. We would compare the result of an expert examination of language, layout, and content of patient information leaflets for a representative number of commonly used medicines with the results of examinations of patient information leaflets by patients who are using these medicines. Our hypothesis was that if a correlation could be established between the results of the expert and the patient examinations it would be acceptable to limit further examinations to the expert tests for most or all of the patient information leaflets currently in use.
METHODS
SELECTION OF LEAFLETS FOR THE STUDY
Since we wanted to recruit participating patients from the pharmacies where they received their medications, the patient information leaflets to be studied were selected from those accompanying commonly used medicines that were available at all pharmacies in Sweden. We obtained a list of 165 medicines that had been prescribed 100000 times or more during 1999 from the Swedish Drug Statistics System (11). From this list we randomly selected 30 medicines for the study and the patient information leaflets accompanying these medicines were examined. Table 1 lists the selected medicines.
THE EXPERT EXAMINATION
Two protocols were set up for the expert examination of the selected patient information leaflets. The protocol (Appendix 1) for the examination of language and layout was based upon the work of Anderson (12), Bjornsson (13), and related standardized assessment techniques previously published (14,15,16). Nine items regarding layout and language were examined. A patient information leaflet that was satisfactory in relation to these items could score a maximum of 25 points. A high number of points would thus indicate good readability and a favorable layout. The selected patient information leaflets were examined by the linguistic expert of the research group (Gunvor Nilsson) and given the appropriate points.
A similar protocol (Appendix 2) was used for the expert examination of the content of the patient information leaflets to determine consistency with Directive 92/27 EEC. An examined patient information leaflet could score a maximum of 25 points, thus indicating that it fulfilled all the requirements of the directive. The patient information leaflets for each of the 30 selected medicines were examined separately by the two pharmacists in the research group (Jan Guslafsson and J. Lars G. Nilsson). When the two raters evaluated a patient information leaflet differently, they discussed the differences to reach a common evaluation. Each patient information leaflet could score a maximum of 50 points (language and content).
THE PATIENT QUESTIONNAIRE
A two-page questionnaire was developed to obtain the patients' opinions of the language, layout, and readability of the patient information leaflet. The questionnaire was to be completed by the patients and returned anonymously to the investigators. We intended to have 40 patients complete the questionnaire for each selected patient information leaflet. We distributed the questionnaires to the patients via 24 randomly selected large and small pharmacies throughout Sweden. The staff of the participating pharmacies received detailed printed and oral information about the study. Each pharmacy was asked to distribute 50 questionnaires: 10 questionnaires for five different medicines. The patient information leaflets for the medicines were distributed among the pharmacies in a random fashion but one of the investigators (Jan Gustafsson) made sure that the selected medicines were commonly sold at each of the pharmacies before the study started. Appendix 3 shows the questionnaire.
PATIENT SELECTION AND INSTRUCTION
Patients who had presented a prescription for one of the selected medicines at one of the participating pharmacies was eligible to be ineluded in the study. The pharmacies were instructed about how to inform the patients about the study and its purpose. Consecutive patients were to be approached until all of the questionnaires had been distributed. When the medicine was handed over by the pharmacist, the patient was asked to participate. The patient was informed that the aim of the study was to improve the information and readability of the leaflet. He/she was, therefore, asked to complete the questionnaire after going home, and to read and use the patient information leaflet when answering the questions. The questionnaire was given only to patients who agreed to participate in the study. After completion of the form, the patient was to mail the questionnaire to the investigators in a prestamped envelope that the researchers supplied along with the questionnaire. The study was approved by the local ethics committee.
DATA HANDLING AND ANALYSIS
Data from the questionnaires as well as from the expert examinations were analyzed using the Statistical Package for Social Sciences (SPSS 9.0). The [chi]^sup 2^-test was used for the statistical analysis. Correlations between the results of experts' and patients' examination of the patient information leaflets were determined using the Excel computer program.
RESULTS
Thirty commonly used medicines were selected and their patient information leaflets were examined by experts and patients as described above. Table 1 shows the medicines selected and the points given in the examinations; the medicines are presented in descending order of expert content scores.
Twenty-four randomly selected pharmacies participated in the study, distributing the patient questionnaires. Two of the originally selected pharmacies were not able to participate and two other pharmacies were randomized to replace them. A total of 1060 forms (88% of the intended 1200 forms) were distributed. For 16 of the 30 medicines fewer than 40 questionnaires were distributed, primarily because some smaller pharmacies did not find the required number of patients with the selected medicines during the patient inclusion period.
The number of questionnaires returned for each medicine was registered. The average response rate was 61%. However, for five of the selected medicines (brands of bendroflumethiazide, dipyramidol, furosemide, paroxetine, and sodium fucidate) fewer than 15 questionnaires were returned, which resulted in response rates of 21% to 44% for these medicines. These patient information leaflets were therefore omitted from further analysis. The results presented in Table 1 are, therefore, based upon 25 different patient information leaflets, which had a response rate of 65%.
THE EXPERT EXAMINATION
In the expert examination of the language (Appendix 1), the patient information leaflets received an average of 16 points (range 10 to 24; Table 1 and Figure 1). None of the patient information leaflets received the maximum 25 points. Eleven patient information leaflets had scores below the average of 16 points. In the corresponding expert examination of the content, the average score was 21 points (range 16 to 24), higher than for the language examination. Twelve of the 25 patient information leaflets had scores below the average of 21 points. Table 1 also presents the total scores for each medicine of the language and content examination. The average total scores for all examinations was 37 out of 50 points (range 30 to 44).
THE PATIENT EXAMINATION
The patient examination based upon the questionnaires (Appendix 3) could score a maximum of 50 points: 25 for layout and language (questions 5 to 10 and 23 to 24) and 25 for content (questions 11 to 21). The test of layout and language resulted in an average score of 23 points (range 19 to 24), which was considerably higher than for the corresponding expert test. All except one leaflet fell within the range of 21 to 24 points. In the content examination, the patients' average score was 15 points (range 10 to 19), which was considerably lower than in the correspending expert test (Figure 2). The average total score of the patient layout, language, and content examination was 38 points (range 32 to 42 points), which is similar to the result for the expert examination.
CORRELATIONS
The scores of the expert versus the patient examinations of content were correlated. The correlation coefficient was 0.669 (p
Table 1 shows the results of expert and patient examinations with the patient information leaflets in descending order of expert content scores. The top 13 leaflets all have expert scores at or above the mean of 21 points and the 12 bottom leaflets have expert scores below that mean. Ten of the 13 top leaflets have patient scores at or above the mean of 15 points, whereas in the bottom group 9 leaflets fall on or below the mean.
Statistical analysis ([chi]2-test) showed that there was a significant correlation between the expert and patient scores (p = 0.0317), indicating that a high expert score for content is similar to a high patient score for the same leaflet, and a low expert score is similar to a low patient score for the same leaflet. Analogous statistical analysis of expert versus patient language examination and the expert versus patient total examination (Table 1) did not provide significant correlations.
DISCUSSION
Directive 92/27 EEC (1) is the gold standard in Europe for patient information leaflets inserted in packages of medicines marketed within the EU. This directive has also been adopted in Swedish legislation. The directive requires that a patient information leaflet have a specified content drawn up in accordance with the Summary of Product Characteristics (SPC) and be: ". . . written in clear and understandable terms for the patient and be clearly legible . . ." The standard for content based upon the SPC is specified in detail, whereas no standard is given for readability and comprehensibility. In this study we have examined four aspects of patient information leaflets:
1. How well, according to experts, the readability and content of randomly selected patient information leaflets adhered to the EU directive,
2. Expert analysis of layout and language to determine readability and comprehensibility,
3. The patients' opinions about the layout and language used in the patient information leaflets, and
4. How well patients could understand the patient information leaflets.
Our data indicate that the results of the expert examinations reflect how easily the patients find and how well they understand the content of the patient information leaflets. The method can, therefore, be used Lo evaluate patient information leaflets in relation to each other. The goal was not, however, to study if the patients see, read, and retain the information in patient information leaflets under normal conditions.
The medicines that were randomly selected from the most commonly prescribed medicines in Sweden in 1999 represent both older and more recently introduced products, and they are used for the treatment of a wide range of dis eases. In the year 2000 when this study was undertaken, the majority, if not all, of standard packaged medicines marketed in Sweden were accompanied by patient information leaflets from the manufacturers.
The patient test involved authentic patients who were taking the selected medicines. These patients could most easily be contacted at the pharmacies. Since consecutive patients were approached at the pharmacies, we consider the patients to be randomly chosen.
In previous studies about the language of patient information leaflets, their readability has been determined using standardized assessment techniques to obtain a grade level indicalion of the reading difficulty (9). We have used essentially the same technique based upon the works by several investigators (12-16), a method that has been validated and widely used in the assessment of written information. The items in the expert protocols (Appendices 1 and 2) arc, as far as possible, congruent with the questions in the patient questionnaire. The scores of the various tests were provided as points to enable a comparison between the results of the protocols and the questionnaire (see below). This is different from previous studies, but it was not expected Lo influence the validity of the tests.
Each requirement of the directive has a corresponding item in the protocol for content examination (Appendix 2) and a corresponding question in the patient questionnaire, except for some points that were not relevant to ask patients. Each item in the protocol for examination of layout and language (Appendix 1) corresponds to analogous items in the questionnaire. The points for each item also correspond as much as possible.
The expert examinations of language and content could result in a maximum of 25 points each, for a total of 50 points (Appendix 1 and 2). An analogous scoring system was used in the patient tests. The language-related questions (5 to 10 and 23 to 24) in the questionnaire could score a maximum of 25 points, which was also the maximum score for the content-related questions (11 to 21, Appendix 3). In that way, the outcome of the expert evaluations could be directly compared to the results of the patient evaluations.
For several items in both the protocols and the questionnaire, graded scores were used. A completely correct answer resulted in 2 or 3 points (Appendix 3), an intermediate answer in 1 or 2 points, and an incorrect or missing answer in O points. Some questionnaire items had a "Don't know" option. If the respondent checked this option, it was scored as O points. There could be two different reasons for this answer. Either the information was in the leaflet and the respondent did not see or understand it, or the information was not in the leaflet and, therefore, the respondent could not answer. In both cases it was a drawback for the leaflet and O points was felt to be the appropriate score.
The expert examination of language showed a wide variability in the scores from 10 to 24 points (Figure 1), with a mean of 16 points. This wide range indicates that the protocol has the capacity to identify both well-written patient information leaflets and those with less suitable language and layout. The corresponding test on patient opinions about layout and language using the questionnaire provided quite a different result, as shown in Figure 1. The average score was as high as 23 points and all leaflets except one fell in the range of 21 to 24 points. This indicates that patients find the layout and language satisfactory and no difficulties in reading and understanding the text were found. However, this result does not correspond to the results of the expert examination, nor with how well the patients understand the content of the patient information leaflets (below). The patients apparently overestimate their ability to use the language when it comes to finding and understanding the information in the patient information leaflets. Thus, the patient questionnaire did not separate well written from less clearly written patient information leaflets.
There was no correlation between the expert language and layout scores and patient scores regarding content (Table 1). Several patient information leaflets scored low (Table 1) in the former examination but both high and low in the patient content test. We interpret this observation to mean that the language and layout may be acceptable for patients who really make an effort to find the answers to the questions on content in the questionnaire. It should be remembered that the patients were actively asked at the pharmacies to read the leaflet accompanying the dispensed medicine and to use the leaflet when completing the questionnaire. It does not provide any information about under what conditions they would read and retain the information in the leaflets, nor how their behavior in that regard would depend upon the quality of the language and layout.
The expert evaluation of content had a narrower distribution of 16 to 24 points (Figure 2) and a higher mean of 21 points than the corresponding language test. The comparatively high average expert score shows that the experts found that most of the information required by the directive was present in the majority of the patient information leaflets.
The patient examination of content gave a mean score of 15 points (range 10 to 19 points). The results of expert and patient examinations of content, thus, appeared as two approximately bell-shaped curves in Figure 2. This indicates that the patient information leaflets have clear shortcomings with regard to the patients' perceptions, since in an ideal situation, we could almost superimpose the two curves. Our interpretation is that the experts are used to reading the type of text found in the patient information leaflets and find all the relevant information, whereas the patients do not have the same training and experience. The reason for this must be that the language and layout of the patient information leaflets is of insufficient quality for patients, although patients expressed almost the opposite opinion when answering the questionnaire. Patients may feel that they understand the medical terms and the long sentences in the leaflets, but when they are asked specific questions regarding the content, they do not always grasp the content and sometimes misunderstand important information. Therefore, there appears to be a clear difference between asking patients their opinions about the language and having the same patients use this language to obtain information. Expert examination of language and layout, therefore, remains an important part in the testing of patient information leaflets.
All possible paired correlations between the results of expert and patient examinations of language and content (ie, between the six columns in Table 1) were tested. The only significant correlation was found between expert and patient examination of content of the patient information leaflets. This shows that a leaflet with strong adherence to the requirements of the directive is likely to have a high rating in the patient examination of the content, and vise versa, a leaflet with low adherence is likely to have a low patient score for the content. This result is not surprising. If a leaflet contains most of the required information and the language and layout are acceptable for patients, this leaflet is bound to score high both in the expert and the patient examinations, although the average score is lower in the latter examination.
The data were further analyzed by comparing high and low scoring patient information leaflets. It was found that patient information leaflets with an expert content score above average correlated significantly with patient information leaflets that had a patient content score above average (the top 13 patient information leaflets of Table 1). This correlation was not found for scores for language or for total scores for language and content.
We consider the results shown here to be representative of how Swedish patients can find and understand relevant information in their patient information leaflets if they are asked to read them carefully. This statement is based upon the large number of returned questionnaires, the response rate of 65%, and randomly selected patient information leaflets, pharmacies, and patients.
We conclude that tests of patient information leaflets could be limited to expert testing of language, layout, and content, since our data and the arguments presented above show that the results of expert tests reflect the results of a corresponding patient test. The protocols presented in Appendices 1 and 2 can be used in the examinations. judging from our data, we propose that a leaflet of good quality should score above the average in both the language/layout test (>16 points) and the content test (>21 points). Such a leaflet is also likely to score high in the patient content test.
Acknowledgment-Grants from Apoteket AB and Stiftelsen Apoteksbolagets fond for forskning och studier i halsoekonomi och socialfarmaci are gratefully acknowledged.
REFERENCES
1. Directive 92/27 EEC. www.europa.eu.int.
2. Royal Pharmaceutical Society. From Compliance to Concordance. Achieving Shared Coals in Medicine Taking. London, United Kingtom: Royal Pharmaceutical Society; 1997.
3. Raynor DK. Patient compliance-the pharmacist's role, lnt J Pharm Prac. 1992;! :126-35.
4. Morris LA, Aikin KJ. The "pharmacokinetics" of patient communications. Drug lnf J. 2001;35: 509-527.
5. Raynor DK, Knapp P. Do patients see, read and retain the new mandatory medicines information leaflets? Pharm]. 2000;264:260-270.
6. Raynor T, Britten N. Why medicines information leaflets do nothing Lo promote concordance. Pharm J. 2001 ;267:120.
7. Collier J. Patient-information leaflets and prescriber competence. Lancet. 1998;352:1724.
8. Coulter A. Evidence based patient information. BrMedj. 1998;317:225-226.
9. Rolland PD. Reading level of drug information printouts: A barrier to effective communication of patient medication information. Drug lnf J. 2000;34:1329-1338.
10. EMEA on testing leaflet readability. Scrip. 2000; 2507:2.
11. Nilsson JLG, Johansson H, Wennbcrg M. Large differences between prescribed and dispensed medicines could indicate undcrtreatment. Drug lnf). 1995;29:1243-1248.
12. Anderson J. Lix and rix: Variations of a little known readability index. I Reading. 1983; 26: 490-496.
13. Bjornsson CH. Lasbarhet. (in Swedish) Stockholm, Sweden: Liber; 1968.
14. Flesch R. A new readability yardstick. ) Appl Psycho/. 1948:32:221-333.
15. Fry EA. A readability formula that saves time. J Reading. 1968;11:513-516.
16. Mc Laughlin HG. SMOG grading-a new readability formula.; Reading. 1969:12;639-646.
Jan Gustafsson, RPh
Apoteket J. J. Berzelius,
Universitetssjukhuset,
Linkoping, Sweden
Sofia Kalvemark, MSc
Apoteket AB,
Stockholm, Sweden
Gunvor Nilssan, PhD
Department of Humanities,
Orebro University,
Orebro, Sweden
J. Lars G. Nilsson, PhD
NEPI Foundation,
Stockholm, Sweden
Reprint Address
Dr. J. Lars G. Nilsson,
NEPI Foundation,
131 88 Stockholm, Sweden
(e-mail: lars.nilsson@nepi.net)
Copyright Drug Information Association 2003
Provided by ProQuest Information and Learning Company. All rights Reserved
Contact
Home
A
Baciim