ICE Digest

ICE Digest

Current IssueArchive Submit Content Contact the EditorReturn to Publications

Overcoming the Threats and Risks of Testing Internationally
By Dennis Maynes and Aimée Rhodes

International expansion of credentialing programs presents exciting opportunities along with significant threats and risks. Exam and program security are critical for delivering good quality growth. Whenever a credentialing program enters a new county, specific security measures should be considered to help preserve the integrity of exams and to generate increased confidence in the quality of test administration. In order to manage threats found in other countries, programs need to quantify operational risks and implement processes that address vulnerabilities, protect against attacks and mitigate risks.

Vulnerabilities are weaknesses in a testing program that can be exploited, or failures that can occur naturally resulting in loss. Threats are potential actions or occurrences resulting from one or more vulnerabilities. Attacks are the realization of threats that may or may not be successful. Risks are possible losses that result from threats and attacks.

Because exam security problems are regularly reported by international media, credentialing programs may be reluctant to embrace international expansion. For example, Jennifer Semko, writing for the Federation Forum, described a Philippines-based test preparation company that received and distributed “a comprehensive compilation of test questions recalled by past test takers” (Semko, fall 2007, “The Story behind the NPTE Score Invalidations”). Many other programs have had similar experiences. On the other hand, the opportunities and advantages of international testing cannot and should not be denied. Our purpose in writing this is to share perspectives and key learnings that will aid your program as you consider international expansion.

Vulnerabilities of Testing Internationally

Whether you are testing at home or abroad, your program will be subject to most of the same vulnerabilities:

  • Individuals can harvest (steal) test items in the same ways.

  • The physical security of testing centers may be breached in the same way.

  • Exams can still be diverted in transit.

International exam security challenges usually result from increased likelihood that vulnerabilities already present will be exploited. Additionally, you may encounter new vulnerabilities. Threats and attacks are more likely to occur when testing in other countries because increased opportunities and motivations to attack are present in more diverse candidate populations (e.g., due to cultural values, varying socioeconomic conditions, ideological differences).

Threats of Testing Internationally

The increased likelihood of exam security threats is driven by three main factors: 1) new vulnerabilities that arise from geographical distances and sociopolitical environments; 2) cultural differences (such as the social desirability of cheating or the acceptance of bribing officials); and 3) high stakes from passing exams.

As an example of increased threats due to high stakes, consider the prevalence of “qiangshou” (literally “gunman”) in China. On April 19, 2014, the Shanghai Daily published a cartoon under the headline: “Hired ‘guns’ in demand to pass tests.” The article described four Chinese individuals who had taken exams for others in more than 25 countries before being caught in South Korea. In some areas of the world, gunmen or proxy test takers are in high demand because passing the test could be the difference between a promising and a bleak future.

Especially pervasive in some countries are electronic devices that enable cheating. Some technically advanced devices use Bluetooth to link audio and video with cellphones. These devices may be hidden almost anywhere (e.g., as a piece of jewelry, in a tie or in glasses). On May 31, 2011, CBC News in British Columbia reported that two men used electronic devices to transmit items from the Medical College Admissions Test to tutors. The Association of Medical Colleges “told police the incident cost them more than $200,000 because the compromised test had to be discontinued and replaced.”

We recommend your organization evaluate the following elements to understand threats in a particular country before you test there:

1. Overall political landscape
             • How stable is the government?
             • Is Civil unrest common?
             • Is regional terrorism a concern?

       2. Firsthand information from other testing programs
             • Have any staff members been threatened or assaulted?
             • What is the experience with fraudulent financial transactions?
             • Have impersonatators interfered with the organization's ability to register test takers?
             • How widespread is cheating on tests?

       3. Travel warnings and advisories that may have been issued for foreign nationals
             • Have travel advisories been issued for the country?
             • Does your organization prohibit travel to the country?
 
       4. Issues with transmissions of exam material
             • Can secure exam content be transferred through customs without being compromised?
             • Can you electronically transmit data (exam content, biomettrics, CCTV video) to and from the country without interference? 

Risks of Testing Internationally

When testing internationally, some losses have higher probabilities and greater harm than others. For example, some Central African countries are dealing with terrorist organizations that have been known to kidnap/kill foreigners.

When exam security attacks are directed by organizations (e.g., Internet-enabled pirates, well-funded commercial enterprises or time-zone cheating syndicates), the probability of loss goes up. Risks are greater because the organizations have more resources for pursuing the attack. Many of these organizations may exploit international testing vulnerabilities (e.g., when the same forms are administered in multiple time zones). An organized group of item harvesters can steal an entire exam efficiently and quickly.

Also, remedies used successfully by U.S.-based credentialing programs may not work in other countries. For example, when foreign websites that distribute harvested exam content disregard Digital Millennium Copyright Act takedown notices, the entire investment in developing the exam may be lost. If the country’s legal system allows for an adequate response to copyright infringement or other attacks on exam security, losses may be mitigated. For example, countries that are signatories to the Berne convention provide copyright-infringement remedies that are similar to those provided by the United States.

You should add these elements to your list as you consider testing internationally

       5. Indices of fraud, corruption and crime
             • Are commercial and Internet fraud prevalent?
             • Is piracy of exam content rampant?
             • Are bribes common and even expected?

      6. Legal protections provided for the protection of intellectual property
             • How quickly are trademarks and copyrights issued?
             • Will copyright registrations of secure tests be honored?
             • Can the rights of copyright holders be enforced?

      7. Ability to conduct investigations
             • Do you have resources that can respond to a breach?
             • Can investigators operate legally in the country?
             • Will police and local law enforcement respond to test theft?

It is important to realize that threats and risks vary between countries. Countries where greater corruption, crime and fraud occur will result in higher probabilities of loss to your organization. Risks may be lessened in countries which provide greater ability to investigate and prosecute instances of test fraud.

Solutions and Countermeasures

The best solutions and countermeasures to international testing threats and attacks are found in strengthening exam security processes. In general, the exam security process can be described using a cycle with four critical steps:

• Protection and deterrence
• Detection
• Decision and response

Protection and deterrence is facilitated by the analysis of vulnerabilities, threats, attacks and risks that have been described above. Some areas of the world may just be too risky for testing. Other areas may be fine when suitable protective steps are taken.

Detection happens through well-trained proctors, site monitoring, Web monitoring and data forensics analysis. Anonymous tip lines also are valuable for this purpose. All are critical for detecting test security incidents and potential breaches.

Decision and response must be pre-planned and ready to deploy in a timely manner. A security incident response plan is essential. Training is critical for ensuring that your organization is prepared and ready to deal promptly with security breaches. It is imperative to enlist the assistance of local experts who can implement your response within each country.

Review and improvement is critical in order to understand the magnitude of risks and losses. Your organization can improve test security by reviewing test security incidents and determining how to improve protection, detection and response.

When making the decision to expand your program into another country, your organization should consider the size of the opportunity and the impact on test takers:

       8. Opportunity and impact of international testing
             • Do you have strategic or economic reasons to begin testing in the country?
             • If entry into the country is delayed, will test takers be excessively penalized?

Whether testing at home or abroad, your program should retain control over the security of the exam administration. You should seriously question the wisdom of testing in areas where this is not possible. The recommended checklist should help you evaluate vulnerabilities, threats, attacks and risks encountered in other countries. This evaluation will guide your selection of specific security measures that help preserve the integrity of exams and generate increased confidence in test administration. After you have taken the proper steps, you will be able to confidently move forward with plans for administering tests internationally.

 

headshotDennis Maynes is the chief scientist at Caveon Test Security. He has pioneered several methods for the statistical detection of potential test fraud, including the use of clusters to detect cheat rings and the use of embedded verification tests to detect braindump users. He has conducted more than 450 data forensics projects for more than 50 organizations, including 11 state departments of education, 10 medical programs and 12 information technology certification programs. Maynes holds a master’s degree in statistics from Brigham Young University.

 

 


headshotAimée Rhodesis the director of examination security at CFA Institute where she and her team are tasked with maintaining the public confidence in CFA-sponsored designations by ensuring that: 1) exam content is secure; 2) no candidate has an unfair advantage over another; 3) only those individuals truly qualified to pass the exam do so; and 4) the risks inherent in exam programs are identified and addressed appropriately. A lawyer by education, prior to joining CFA Institute Rhodes was the director of test security at ACT.