100% cloud-native, Lighthouse builds on eMetric’s proven high-stakes assessment delivery platform, iTester, and its flagship assessment data reporting and analytics platform, Data Interaction, to provide test providers in any industry the most flexible, robust, end-to-end assessment management solution available today.
Lighthouse offers flexible implementation options along with a full suite of modules for assessment development, administration, digital proctoring*, scoring, and reporting. It delivers a seamless and consistent experience to manage multiple assessment needs ranging from simple classroom scenarios to high-stakes accountability and certification programs. Lighthouse provides an easy, efficient, and transparent workflow for test providers to configure their assessment system with integrated management of test content, identity management, data management, and professional services.
Delivering an anxiety-free test-taking experience to all test takers is a responsibility we take very seriously. We designed our delivery platform to provide industry leading resilience to failure.
Modern AI capabilities detect student-test engagement, aberrant behavior in real-time, automatically score student responses, and predict student growth to help tailor instruction.
Leveraging our native-cloud microservices based architecture, we can cost-effectively cater to any program at any scale.
We offer the most comprehensive standards coverage across the platform encompassing systems, content, data, processes, and accessibility.
Delivering an anxiety-free test-taking experience to all test takers is a responsibility we take very seriously. We designed our test delivery platform to provide industry leading resilience to failure. Our technologies provide automated self-healing, retry, and circuit-breaking capabilities to tackle the problems of unreliable networks and scenarios that arise daily in schools and classrooms.
We built specialized bots featuring state-of-the-art artificial intelligence (AI) and machine learning (ML) methods to continuously test our delivery platform. The AI/ML models utilize millions of click streams featuring real student test behavior. Our bots work 24/7 to find issues before they happen.
We fine-tuned and hardened our platform, operations, and infrastructure to scale for the high-concurrency, thundering-herd, and flash-flood traffic problems of large-scale online test delivery. We routinely test for 2x the anticipated capacity of concurrent test takers during seasonal spikes and induce rigorous test taker activity ranging up to 5x real user activity to stress the system, sub-systems, and the underlying infrastructure components.
No single component can guarantee 100% uptime and even the most expensive hardware eventually fails. That’s why we designed a regionally-redundant cloud infrastructure where individual components can fail without affecting the entire system.
Powerful machine learning algorithms coupled with our real-time streaming analytics deliver insightful notifications regarding student-test engagement to proctors and test administrators.
Standardized tests require the highest-level of integrity and security standards to ensure fairness and validity of the assessments. Early detection and prevention of abuse is critical to maintain validity and fairness to all test takers. We leverage state-of-the-art advances in AI/ML to detect aberrant test-taking performance and behavior.
We built state-of-the-art deep neural networks to score short answers and essays featuring models that require very little learning.
We built a powerful regression-tree based approach to modeling student performance and have thoughtfully embedded these insights in an educator-friendly reporting and analytics platform. These insights are dynamic and are a valuable asset to cater instructional strategies and interventions to students and cohorts that are in need.
Our platform is the ultimate destination for administering custom large-scale high-stakes assessment programs. We have industry leading security, scalability, and reliability to meet the demanding needs of summative assessment programs. We can handle states of any size and assessment programs of any complexity.
Administering your interim or formative assessment programs has never been easier. Our state-of-the-art reporting and analytics platform in conjunction with the integrations available for instructional resources will make this your preferred choice for all your assessment management needs.
The assessment platform has support for program specific workflows and support for tools (speech capture for spoken tests) that are unique to certain assessment programs such as ELL and Special Education assessment programs. These workflows and tool options are driven by configurations specific to the assessment programs selected for administration.
We have invested in making our platforms open and interoperable with industry standards such as SIF, QTI, APIP, and OneRoster for test content and data systems and WAI-ARIA for accessibility. We also have a robust and open API ecosystem for integrating solutions and services from vendors and partners that enables us to ultimately offer an integrated and seamless experience to our customers.
We understand that a ‘one size fits all’ approach doesn’t always scale. Standards can have implementation gaps that require varying levels of customization. To address this, we built processes and tools that make this seamless and practical, while maintaining compatibility and compliance with standards.
We have industry leading support for assistive technologies and comprehensive coverage of accessibility standards to ensure that our platform is accessible for all test takers. We support a broad range of assistive technologies and accessibility features including text-to-speech (multiple languages), Refreshable Braille, ASL, and screen readers.
Oklahoma
Oklahoma has used iTester for high-stakes summative testing since 2014. Tests with innovative technology enhanced items are authored, administered, automatically scored (except writing prompts), and reported within the single sign-on Oklahoma Portal.
Client Spotlight: Oklahoma
New Mexico
eMetric has provided online assessment testing services for the New Mexico Standards Based Assessment (NMSBA) for Science since fall 2014. Both practice test and high-stakes testing services are provided using eMetric’s online testing platform, iTester.
Client Spotlight: New Mexico
Pennsylvania
eMetric has provided online assessment reporting services for Pennsylvania since 2004 via Data Interaction. In addition to providing a data query tool, eMetric developed and hosts PSSA Summary Reports and Parent Letters within Data Interaction and provides a publically accessible website for federal accountability reporting.
Client Spotlight: Pennsylvania
Alaska
eMetric has provided online assessment reporting services for Alaska since 2010 via Data Interaction. Alaska users can access a variety of assessment results from a single portal including Standards Based Assessments, High School Graduation Qualifying Exam, Alternate, English Language Proficiency, and TerraNova assessments.
Client Spotlight: Alaska
Nevada
eMetric has provided online reporting services for multiple statewide assessment programs via
Data Interaction since 2002. Services include providing a secure data query tool, pre-defined
reports, and an online interface for state and district users to review and/or edit data. Since
2016, eMetric provided a secure site for Nevada Smarter Balanced Assessment Program. Since
2012, eMetric has also served as the service provider for the Nevada Longitudinal Data System
which includes the reporting services for the Nevada School Performance Framework, the
Nevada Growth Model, and the Nevada Report Card as well as data management services for
the state data system.
Client Spotlight: Nevada
Maryland
iTester has powered the Maryland High School Assessments (HSA) in Science (Biology) and Government for high school grades 9 through 12 since 2017.
Client Spotlight: Maryland
Tennessee
Data Interaction powers the Tennessee Department of Education’s assessment reporting
system providing educators and parents with online, dynamic access to current and historical
performance data for state assessment programs.
Client Spotlight: Tennessee
Create and deliver best-in-class solutions for assessment creation, delivery, and reporting that empower educators and decision-makers with timely insight into student knowledge and performance.
eMetric LLC is a privately held, for-profit company headquartered in San Antonio, Texas. Founded in 2000 by Dr. Huixing Tang, eMetric is a leading provider of technology platforms for the K-12 assessment industry. We develop and integrate test authoring, administration, delivery, scoring, and reporting modules for online assessment. While we have developed these modules to integrate seamlessly as end-to-end solutions for districts and schools, our core strength is creating and licensing our robust, reliable online assessment administration and reporting platforms to leading K-12 assessment publishers and states for high-stakes assessment programs. These platforms have been used successfully for more than a decade to service many statewide summative assessment programs.
eMetric is focused on providing impeccable assessment technologies based on uncompromised expertise in measurement, data science, and technology. Our products, beginning with the initial online assessment reporting application in 2000 to the recent releases of our online assessment delivery and reporting platforms display a history of leveraging the latest technology to advance the assessment industry. The latest versions of our platforms are greatly enhanced by our solid research in machine learning and deep learning with neural-networks and microservice oriented cloud native technology. We are proud of our reputation for being innovative and always on the cutting edge of technology.
The eMetric technical team is comprised of experienced statisticians and data scientists, software engineers, system architects, and UI designers, all well-versed in current development languages and methodologies. eMetric’s capabilities are further strengthened by a talented team of quality assurance analysts, project managers, business analysts, and client support specialists. These teams work collaboratively to ensure the highest levels of reliability, usability, and client satisfaction for every contract.
Committed to continuous improvement, eMetric continues to enrich our core products and seek innovative ways to meet the online assessment and reporting needs of our clients. eMetric’s portfolio of online assessment and reporting solutions revolve, and evolve, around the company’s goal to empower educators and decision-makers with timely insight into student performance.
|
eMetric San Antonio |
Click below or call us at (877) 829-7769
We're ready to answer any questions you have about our products and services.
Need help? We're here.
We're dedicated to developing the most robust, innovative, and powerful technology to transform the educational experience. We're looking for other smart, motivated, developers and innovators to join our team!
To view our current open positions, click here!
Easily produce customized, actionable reports to pinpoint strengths and areas for improvement with Data Interaction's intuitive design and rich capabilities.
Quickly identify students for intervention using student growth and next-year forecast data based on regression-tree based approach to modeling student performance.
Interactively build and interpret groups of students based on the combination of user-selected options, generated on the fly using modern machine learning methodologies.
Data Interaction's rich suite of analysis tools empowers educators to make informed decisions. The intuitive interface provides quick, easy access to review summary data from a broad perspective, drill down to specific subgroups, or zoom in to the student level to identify a student's individual strengths and weaknesses.
Data Interaction is proven to support both secure and public reporting sites, including state accountability sites and parent portals.
A secure site allows role-based access for users at the state, district, and school levels with privileges defined by the site administrator. Summary, roster, and individual student results can be viewed in an interactive environment with easy-to-use options for filtering, searching, analysis, and ad hoc querying.
A publicly accessible site allows for quick, interactive access to customizable summary reports with easy-to-use options for aggregations, disaggregations, filtering, and ad hoc queries, as well as pre-defined reports.
Public sites feature a range of views and reports to allow analysis across years at the state, district, and school levels. Each view and report can be customized to include or exclude fields and attributes to meet a client's specific needs.
Data Interaction has also proven to successfully report non-assessment measures via accountability and school report card sites.
A parent/student portal provides exclusive, on demand access to individual student assessment performance data from throughout the student's academic career. Parent/student sites feature a responsive design, ensuring that parents and students have an optimized view of important student assessment performance data, whether they are accessing from a desktop, laptop, tablet, or smartphone.
eMetric's design philosophy embodies a mobile-first approach for all components. The result is a design directed at mobile devices, rather than a watered-down experience of the desktop platform. This provides information for users when and where they need it, which is often not sitting at their desks behind a PC.
iTester is a suite of robust applications for item and test authoring, scheduling, administration, test delivery, and machine scoring of online assessments. iTester was designed to support the specific needs of high-stakes testing while providing the flexibility to manage and deliver multiple testing programs (including formative, interim, benchmark, and summative assessments) in a single environment.
Today's educators and students interact, learn, and play in an increasingly digital environment. They are accustomed to engaging and straightforward interfaces that naturally facilitate exchange of information and knowledge. Our thoughtfully designed online testing platform provides an authentic experience that keeps users engaged and better mimics their daily activities. iTester offers the reliability, scalability, feature set, and security to ensure technology is an enabler, not a distraction.
The iTester platform has earned a reputation as one of the most technologically advanced testing platforms available and has functioned flawlessly since it was initially released in 2008. Our team has extensive experience working with states, districts, and test publishers across the country.
iTester’s intuitive WYSIWYG interface provides item authors the ability to create any item they can imagine ranging from simple multiple choice items to advanced, multi-part technology enhanced items. Item writers from major test publishers, state education agencies, and school districts have successfully used iTester to construct items, forms, and author and publish tests for summative high-stakes assessment programs as well as local formative and benchmark assessment programs.
iTester’s flexible authoring module simplifies and facilitates collaboration with our clients and partners. Pre-existing item banks can be uploaded using iTester’s QTI importer tool. Once a QTI item package is uploaded to iTester, imported items can be previewed, edited, and finalized. Our clients and partners have found iTester’s QTI capabilities to substantially lessen the burden associated with changing assessment vendors.
The iTester assessment management portal provides role-based, single sign-on access to our item and test authoring component, data management functionality, and test scheduling, as well as the tools and resources IT personnel need to prepare for a successful test administration. Add on our dynamic reporting and data analytics platform, Data Interaction, and authorized users will also have single sign-on access to interactive summary, roster, and individual student reports as well. Our comprehensive, intuitive workflow has been carefully crafted to meet the rigorous and demanding requirements of high-stakes testing, while minimizing the burden on district and school test administrators.
iTester provides a comprehensive suite of tools and accommodations to ensure a fair, accessible test-taking experience for all students. From embedded tools and accommodations to system level assistive technology compatibility, your students are at the heart of every design decision we make.
iTester is designed to provide a consistent, intuitive user experience regardless of test-taking device, including iPads, Chromebooks, and Windows-based tablets.
High-stakes assessment requires technology platforms to be invariably available and reliable. eMetric has employed ‘no single point of failure’ design principles in the development of iTester resulting in an impeccable track record for high-stakes test delivery. We know that ultimately product quality is paramount so we strictly follow a comprehensive approach to quality assurance to ensure we exceed our client’s expectations.
We understand the realities that exist in schools in regards to available technology, bandwidth, and technical support. Consequently, the iTester platform has been fine-tuned to account for a host of unique technology environments and constraints that exist in many schools.
Advance encryption and failbacks actively safeguard the confidentiality and integrity of test content, student information, and student responses.
Our in-house research and development efforts have led to a number of product innovations as well as innovative processes to ensure the quality and reliability of our platforms. For example, our quality engineers created an intelligent QA Bot capable of using insights from historical click streams to produce simulations of student test-taking behaviors. These simulations are used to uncover software errors during quality engineering checks, preventing disruptions to live testing!
Our QA Bot is just one way we are harnessing the potential of deep learning to forward the state of the art in online testing and reporting. Visit our Research page to learn more about our current explorations!
eMetric’s iAuthor item authoring and banking system provides intuitive, sophisticated tools and workflows for creating innovative items and engaging assessments. Item writers from major test publishers, state education agencies, and school districts have successfully used iAuthor to construct items, forms, and tests for both statewide summative assessment programs and local assessment programs.
iAuthor’s streamlined processes enable test publishers, subject matter experts, and educators to easily author and import items, construct forms, and construct and publish tests. With a library of over 30 interaction types, iAuthor empowers item writers to create innovative items and rich assessments that more effectively test critical thinking and problem-solving skills, allowing students to fully showcase their knowledge and skills.
Lighthouse is built on a scalable microservices based architecture that offers instant elasticity and flexibility. Highly configurable, Lighthouse suits a variety of assessment program needs from low-stakes classroom testing to high-stakes state-wide summative testing using the same platform. This reduces the time spent on training administrators, streamlines the development and production of materials and documentation, and most importantly, allows students to take high-stakes tests in the same, familiar environment in which they take their classroom tests. Lighthouse isn’t just another technology platform; it’s the assessment management platform that will propel your assessment program into the future.
Learning doesn’t stop beyond the walls of a school, and neither should your assessments. Today more than ever schools and organizations need affordable, flexible options for ensuring the security and integrity of online testing. eMetric’s iTester assessment management platform now features a powerful remote proctoring module, TestStream™, enabling educators and students to assess learning anytime, from anywhere.
TestStream is a suite of secure test delivery technologies that collects, tracks, and monitors test security events using a combination of logs, screen recordings, and live camera feeds to enforce test security. These technologies work in concert to ensure that a test is delivered to an authorized test taker in a secure environment and location and administered using a consistently enforced test security protocol by a remote proctor.
eMetric's TestStream proctoring solution is enhanced with our proven test taker activity tracking solution, ClickStream. ClickStream’s machine learning algorithms provide advanced alerting and monitoring features to effectively enforce test security protocols and provide insightful data analytics.
TestStream offers both live and automated remote proctoring solutions, providing cost-effective, easy to use options for high-stakes and lower-stakes exams.
ClickStream |
ScreenStream |
LiveStream |
|---|---|---|
| Purpose-built synthetic event tracking of test taker's interactions | Real-time video stream of the test taker's screen | Real-time two-way interactive video/audio stream between test taker and proctor |
| Available by default for all tests delivered through the iTester Test Delivery Platform | Optional (program and test level configuration) | Optional (program and test level configuration) |
|
|
|
| Applicable for all assessment scenarios | Ideal for medium-stakes assessments where alternate forms of test taker identity are acceptable | Ideal for high-stakes assessments where real-time surveillance of test taker identity and surroundings is critical |
| Available across all devices and form factors supported by iTester | Available across all devices and form factors supported by iTester | Compatible microphone and webcam required |
eMetric is committed to transforming the educational experience for both educators and students. We're looking for other talented, motivated individuals to join us in creating the cutting-edge software solutions that will realize our vision.
We take pride in our products and know they have an enormous impact on thousands of students every day. We know that best-in-class solutions demand best-in-class talent and we are committed to surrounding ourselves with smart, highly-skilled problem solvers who thrive both independently and as team members.
Our employees are our greatest asset and we are committed to making sure you love coming to work every day! In addition to a casual, friendly and fun work environment, all employees receive:
eMetric is a San Antonio based technology platforms provider for K-12 education seeking a Data Scientist to join our Data Science group. Our current initiatives involve NLP classification, modeling and forecasting performance trajectories, and clustering high dimensional school and student data. We know that best-in-class solutions demand best-in-class talent. We love surrounding ourselves with smart, highly-skilled problem solvers and researchers who thrive independently and as team members.
Primary Responsibilities
Skills and Qualitifications
eMetric is looking for an exceptional Software Engineer to develop its cutting-edge products, including online assessment platforms and business intelligence tools for K-12 education.
eMetric is a San Antonio based technology platforms provider for K-12 education. We pride ourselves on finding innovating and unique ways to apply technology to the fast- growing EdTech industry. We deal primarily in high-stakes student assessments which demand the highest rigors of software engineering to ensure millions of students are able to test online successfully and get their results back quickly and easily. We offer a dynamic and team-oriented environment with ample opportunities for significant individual contributions to our products.
As a Software Engineer, you will utilize the .NET technology stack with C# to build and collaborate on the delivery of solutions on the Microsoft Azure cloud. We use a microservices-based architecture delivered on the Kubernetes container-orchestration system. You will also leverage technologies like JavaScript, Node.js, React, and SQL Server to fulfill various product and platform needs. Our teams use the Scaled Agile Framework (SAFe) to empower an agile development process that puts the focus on creating value.
Requirements:
Preferred Qualifications:
Duties & Responsibilities:
We are committed to surrounding ourselves with smart, highly-skilled problem solvers who thrive both independently and as team members. Engineering excellence is the norm and encouraged. This is a challenging, yet rewarding domain where you can apply the latest technologies to help empower students and education across the country. Our employees are our greatest asset and we are dedicated to making sure you love coming to work every day! In addition to an excellent work environment, all employees receive medical and dental insurance, long and short-term disability insurance, paid vacation and holidays, flexible hours, fitness club membership reimbursement, amenity-rich offices, and a casual work environment.
eMetric is a San Antonio based technology platforms provider for K-12 education seeking a Database Developer. We know that best-in-class solutions demand best-in-class talent. We love surrounding ourselves with smart, highly-skilled problem solvers who thrive independently and as team members
Primary Responsibilities include:
Qualifications
We take pride in our products and know they have an enormous impact on thousands of students every day. We know that best-in-class solutions demand best-in-class talent and we are committed to surrounding ourselves with smart, highly-skilled problem solvers who thrive both independently and as team members. Our employees are our greatest asset and we are committed to making sure you love coming to work every day! In addition to an excellent work environment, all employees receive medical and dental insurance, long and short term disability insurance, paid vacation and holidays, flexible hours, fitness club membership reimbursement, amenity-rich offices, and a casual work environment.
eMetric's powerful and intuitive platforms are built using the latest technology with a resolute commitment to quality. Quality is a characteristic that is often difficult to assess until after an investment decision is made. Impressive feature sets and competitive pricing are enticing, but ultimately product quality prevails. Our track record speaks volumes to eMetric's dependability and commitment to excellence. We consider our record on contract performance to be superb and have earned our reputation as an easy to work with, technically advanced, dependable partner.
We have extensive experience in the K-12 market having served the needs of multiple states and districts for over 20 years. We follow education trends and stay abreast of changes in legislation. We know our clients are our greatest source of industry insight and we benefit from strong relationships with all of our clients. We regularly solicit feedback both formally and informally from clients and end users. Our extensive technology capabilities, experience in K-12 assessment, proven collaborative abilities, and skilled team provide our clients with a talented, committed partner.
Here is what our clients say about us:
The Maryland High School Assessment for Government and Maryland Integrated Science Assessments (MISA) has been delivered using iTester since 2017. Students take their tests on desktops and laptops, as well Apple tablets and Chromebooks. All tests are authored, administered, and delivered using iTester.
eMetric has provided online reporting services for multiple statewide assessment programs in Nevada via Data Interaction since 2002. From 2002 through 2006, eMetric provided a public website for dynamic access to the assessment results for Nevada Criterion Referenced Test (CRT), Nevada High School Proficiency Exam (HSPE), and Nevada Norm Referenced Test (NRT). From 2006 to 2017, eMetric provided a secure website for the state's writing assessment program.
Services include providing a secure data query tool, pre-defined reports, and an online interface for state and district users to review and/or edit data. Since 2016, eMetric has provided a secure site for the Nevada Smarter Balanced Assessment Program.
Since 2012, eMetric has also served as the service provider for the Nevada Longitudinal Data System which includes the reporting services for the Nevada School Performance Framework, the Nevada Growth Model, and the Nevada Report Card as well as data management services for the state data system.
eMetric has provided online assessment testing services for the New Mexico Standards Based Assessment (NMSBA) for Science since fall 2014. Both practice test and high-stakes testing services are provided using eMetric’s online testing platform, iTester.
Oklahoma has used iTester for high-stakes summative testing since 2014. Tests with innovative technology enhanced items are authored, administered, automatically scored (except writing prompts), and reported within the single sign-on Oklahoma Portal.
eMetric has provided online assessment reporting services for Pennsylvania since 2004 via Data Interaction. In addition to providing a data query tool, eMetric developed and hosts PSSA Summary Reports and Parent Letters within Data Interaction and provides a publically accessible website for federal accountability reporting.
Data Interaction powers the Tennessee Department of Education’s assessment reporting system providing educators and parents with online, dynamic access to current and historical performance data for state assessment programs. The reporting system consists of two major portals: a parent/student portal and a secure educator portal available to district, school, and teacher level accounts.
eMetric's Deep Learning Essay Scoring (DLES) engine uses the latest innovations in natural language processing techniques to formulate contextual representations of words, sentences, and paragraphs to provide reliable and accurate essay scores.
The DLES engine is based off of models of language trained with huge corpuses of text, such as all of Wikipedia and hundreds of thousands of books. The DLES engine fine-tunes its model of language to understand how to score student essays through a training process involving hundreds of thousands of student essays.
The DLES engine is based off of modern deep learning artificial intelligence techniques which enable it to understand language in more nuanced ways than traditional automated essay scoring (AES) techniques, which often rely on simple statistics such as word counts, counting similar words between essays, and number of commas. Our initial research shows that the DLES engine can already outperform traditional AES models, and we are actively developing and improving our system with new training data and new approaches.
For 20 years, eMetric has been a leading provider of digital solutions for high-stakes and large-scale K-12 assessment delivery and reporting.
We recognize there is an enormous potential for artificial intelligence (AI), data analytics, and machine learning methods to provide practical improvements in digital education, assessment delivery, scoring, and reporting.
Remaining true to our identity as a cutting-edge technology solution provider, we are currently committed to an aggressive research agenda focused on finding novel applications for AI to enhance our products and offerings.
Our talented team of psychometricians, researchers, and engineers is currently investigating the following promising applications of deep learning:
National Council on Measurement in Education (NCME) annual meeting in Chicago
Saturday, April 15, 2023
In this study, several approaches towards predicting behavior in a test environment are analyzed with the purpose of quantifying how typical (or atypical) a student’s behaviors are in a test context, providing a summary measure of a test-taker’s behaviors, allowing for further investigation of any test-takers who are displaying atypical behavior patterns. The proposed behavior models include architectures such as the Long Short-Term Memory (LSTM) network, Recurrent Neural Networks (RNN), and an n-gram approach. The characteristics of MAI indexes and correlations between MAI results and other existing statistics for detecting aberrant test-taking behaviors are discussed.
LSTM Sequence Modeling to Summarize Test-taking Behaviors - NCME 2023 (pdf)
Psychological Test and Assessment Modeling
Volume 65, 2023 (1), 76-124
A new statistic, known as the Model Agreement Index (MAI), is introduced. MAI indicates how “typical” test taking behavior is, allowing for the potential to flag atypical behavior, utilizing clickstream behaviors collected during test taking. In the framework, a behavior model is first trained to emulate how typical examinees act during a test based on recorded individual examinee clickstream data. A Long Short-Term Memory (LSTM) sequence model is used, as they have shown superior predictive performance in sequence modeling tasks in other applications. The study details the processes needed to train the LSTM along with a comparison between the LSTM and a “most common next action” baseline model.
https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam_2023-1/PTAM__1-2023_4_kor.pdf
Psychological Test and Assessment Modeling, Volume 64, 2022 (4), 445-470
Published December 12, 2022
This study discusses aspects of using Gradient Boosted Models (GBMs) in computing growth percentiles by 1) illustrating the effects of different hyperparameters on model fit, 2) comparing GBM and QR-based SGP agreement across different sets of predictors, 3) using an interpretability method, SHAP (SHapley Additive exPlanations), to show the impact of each predictor on the predictions of the GBM model, and 4) analyzing the effect of sample size on GBM prediction accuracy.
https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam_2022-4/PTAM_2022-4_4.pdf
The National Council on Measurement in Education (NCME) Conference
June 8, 2021
This presentation highlighted a demonstration of using clustering to group schools together by various traits. K-means, hierarchical, and density-based clustering on real testing data from hundreds of elementary schools from a single state were used. A “visual clustering” approach was demonstrated to allow stakeholders to engage with the clustering in real-time.
National Council on Measurement in Education (NCME) annual meeting in Toronto, Canada
Monday, April 8, 2019, 12:20-1:50 pm
In education, big data analytics methods have become increasingly popular (Romero & Ventura, 2010). This article illustrates how we use XGBoost regression trees for predicting students’ future performance in state summative tests. Bayesian networks and linear regression model are applied for comparison. Results show that XGBoost regression trees perform the best, with higher prediction accuracy and computation efficiency. The XGBoost regression tree also works better with incomplete data sets.
Forecasting students’ future academic performance using big data analytics (PDF)
National Council on Measurement in Education (NCME) annual meeting in Toronto, Canada
Sunday, April 7, 2019, 12:10-1:40 pm
This study compares percentile rank residuals using an XGBoost regression tree model to quantile regression based SGP. Results indicate that with default hyperparameters, the XGB tree based approach can exactly replicate standard SGP, and that the XGB method may be further tuned to potentially predict more accurately.
Applying Gradient Boosted Regression Trees to Produce Growth Percentiles (PDF)
March 18, 2019: 12:30 pm – 2:00 pm
This presentation presents IQBot, an intelligent test-taking bot. IQBot is intended for use by test vendors responsible for maintaining test delivery engines. Ensuring that high-stakes assessment software works in all circumstances is mission critical for testing companies. IQBot mimics student behaviors in taking computer-based tests, and is intended to cover the breadth of testing functionality, including accommodation usage, tool usage, using navigation systems and review panels, and so on. Thousands of IQBots can be launched to provide comprehensive system-testing even on test forms that have not yet been deployed in the wild. IQBot will automatically log any system errors and provide checks such as ensuring that the answers entered match the answers stored in the database. This presentation will show how the AI-enhanced IQBot, utilizing neural network intelligence, can system-test software in real time by spawning multiple IQBot processes along with actual feedback generated.
March 18, 2019: 4:00 pm – 5:00 pm
ATP 2019 in Orlando, Florida
Detecting cheaters in assessment contexts is vital to ensuring test-score validity. In this presentation, the use of clickstream clustering techniques is presented as a means to generalize different behavioral patterns exhibited by students. Once a student is assigned to a behavioral pattern cluster, their performance can be compared to the average performance of that respective cluster. Aberrantly high scorers can be identified for further investigation. Existing cheating detection algorithms often rely on response time patterns, which can help mitigate some facets of cheating. This presentation will showcase the potential to cluster students based on tool usage, navigation patterns, answer changing, and other behaviors logged in the clickstream data. The presentation will give details on the types of clustering algorithms used, examples of how to identify an aberrant score based on clustering, and future considerations for data-science driven approaches for cheating detection.
ATP 2019 in Orlando, Florida
Summative test scores are snapshots of a student's ability level in a particular subject and at a particular point in time. In an ideal world, students, teachers, and parents will use these test score snapshots to help guide instruction in the future. In the modern age of data analytics and predictive analytics, there is always the question of whether further insight can be derived from the data that is already available. With additional scores and metrics derived from analytics, stakeholders can be guided to provide the right type of instructional intervention, either at the personal or the curriculum levels. This presentation provides an initial investigation into the potential of using predictive analytics techniques to predict future student performance based on longitudinal history information, such as past scores and past demographic information. Additional factors are also investigated, such as a statistical model's ability to handle missing data.
National Council on Measurement in Education (NCME) annual meeting in New York, USA. Saturday, April 14, 2018, 4:06-6:05 pm
Description of an easy-to-compute statistic for detecting examinee’s aberrant response times in large-scale assessments. A simulation study and an empirical study were conducted to evaluate its performance. Results show that the new statistic performed equivalently well to van der Linden & Guo’s (2008) Bayesian procedure, and reduced computation burden monumentally.
Easy to Compute Response Times Based Satistics for Detecting Aberrant Behaviors of Test-Takers (PDF)
Before using this website, please read this Privacy Policy for this website carefully. By using this website and its services, you acknowledge that you understand, accept, and agree to the terms of this Privacy Policy for this website. If you have any questions concerning this Privacy Policy for this website, please contact eMetric at (877) 829-7769.
This Privacy Policy applies to this website and is not applicable to the software applications we may provide to your educational institutions. Please note that the collection, input, use, retention, disposal, and disclosure of any private information from our software applications is controlled by your student’s educational institution.
When you access or use this website, we may collect two types of information: (1) personally identifiable information; and (2) information that is not personally identifiable. An example of personally identifiable information that may be collected on this website includes your name and email address if provided via this website for sales or support. We do not collect any personally identifiable information about you unless it is voluntarily provided. Any personal information provided to eMetric is strictly for internal purposes only.
We may collect non-personally identifiable information about your visit when you access our website. The non-personally identifiable information we collect includes the domain name of the website providing you with Internet access and the Internet protocol (IP) address used to connect your computer to the Internet. The IP address is usually associated with the place from which you enter the Internet, such as your Internet Service Provider or your school district. We may also collect your browser type and version, the operating system your computer is running, and other website visit data such as the pages you view, and/or the information you search for.
eMetric never sells, trades, shares, or discloses personally identifiable information or non- personally identifiable information to other companies. We do not permit behavioral targeting of advertisements.
Except for the limited exceptions set forth below, eMetric will not share with, or disclose to, third parties, personally identifiable information collected through this website. eMetric may, however, disclose personally identifiable information:
Non-personally identifiable aggregated information collected through this website may be used for quality assurance purposes and may be used to increase the functionality and user- friendliness of our website.
This website may also use cookies to measure traffic patterns, personalize content, and control security. A cookie is a small file placed on the hard drive of your computing device through your browser or app. You may refuse to accept browser cookies by activating the appropriate setting on your browser. This website may collect IP addresses, the associated URLs, domain types, the browser type used to access the website, the country, state, and telephone area code where the users' Internet service provider's servers are located, the pages of this website that users viewed during their visit and any search terms entered on the website. Collection of IP addresses is generally for system administration purposes, to monitor the level of activity on the website, and for security reasons.
This website is not directed to persons under the age of 13. eMetric does not knowingly collect personal information from children under 13 through this website. Students under the age of 13 are specifically requested to not provide any personal information through this website. If you become aware that a child has provided personally identifiable information through this website, please contact eMetric at (877) 829-7769.
The security of your personal information is very important to us. eMetric uses commercially acceptable means to protect any personal information collected via this website. However, despite our best efforts, no method of transmission over the Internet is completely secure. We cannot guarantee its absolute security. eMetric does not assume liability for any other party’s use of your personally identifiable information.
This website may include links to third party websites. When you link to another site outside of the emetric.net domain, this Privacy Policy does not apply. When you link to another website you are subject to the privacy policy of that new site.
This Privacy Policy for this website may be revised from time to time through an updated posting. If material changes are made to this Privacy Policy, the changes will be posted on this website to alert you. You should, therefore, check this Privacy Policy periodically. Revisions are effective upon posting and your continued use of this site following the posting of such revisions will indicate your acceptance of such revisions.
If you have questions or concerns about this Privacy Policy, please contact us by telephone at (877) 829-7769. If you want us to respond to your comment or question, please provide your contact information in your message.
Antes de utilizar este sitio web, por favor lea atentamente la Política de Privacidad para el mismo. Al utilizar este sitio web y sus servicios, usted confirma que entiende, acepta y está de acuerdo con los términos de la Política de Privacidad para este sitio web. Si tiene alguna pregunta concerniente a la Política de Privacidad para este sitio web, por favor contacte a eMetric al (877) 829-7769.
Esta Política de Privacidad aplica a este sitio web, y no a las aplicaciones de software que podamos proporcionar a sus instituciones educativas. Por favor tome nota que la recopilación, ingreso, utilización, retención, eliminación y divulgación de cualquier información privada de nuestras aplicaciones de software es controlada por la institución educativa de su estudiante.
Podemos recopilar dos tipos de información cuando usted ingresa a este sitio web o lo utiliza: (1) información de carácter personal; e (2) información de carácter no personal. Un ejemplo de información de carácter personal que puede ser recabada en este sitio web incluye su nombre y dirección de correo electrónico, si fueron proporcionados por medio de este sitio web para ventas o asistencia. No recopilamos ninguna información de carácter personal sobre usted, a menos que sea proporcionada voluntariamente. Cualquier información personal proporcionada a eMetric es exclusivamente para fines internos.
Podemos recabar información de carácter no personal sobre su visita cuando usted ingrese a nuestro sitio web. La información de carácter no personal que recopilamos incluye el nombre de dominio del sitio web que le proporciona acceso a Internet, y la dirección del protocolo de Internet (IP) utilizada para conectar su computadora a Internet. La dirección de IP está asociada usualmente al lugar desde el cual ingresó a Internet, tal como su Proveedor de Servicios de Internet (ISP) o su distrito escolar. Podemos también recopilar su tipo de navegador, su versión de navegador, el sistema operativo que utiliza su computadora, y otros datos de la visita al sitio web, como las páginas que usted consulta, y/o la información que busca.
eMetric nunca vende, comercia, comparte o divulga información de carácter personal o información de carácter no personal con otras empresas. No permitimos la publicidad de segmentación conductual.
eMetric no compartirá ni divulgará a terceros información de carácter personal recabada a través de este sitio web, excepto bajo las excepciones limitadas mencionadas a continuación. Sin embargo, eMetric podría divulgar información de carácter personal:
La información agregada de carácter no personal recabada a través de este sitio web puede ser utilizada para fines de control de calidad, y para aumentar la funcionalidad y facilidad de uso de nuestro sitio web.
Este sitio web también puede utilizar cookies para medir patrones de tráfico, para personalizar contenido, y para controlar seguridad. Las cookies son archivos pequeños que se instalan en el disco duro de su dispositivo informático a través de su navegador o aplicación (app). Puede rehusarse a aceptar cookies del explorador activando el configurador apropiado en su navegador. Este sitio web puede recabar direcciones de IP, los URL (Localizadores Uniformes de Recursos) asociados, los tipos de dominio, los tipos de navegador utilizados para ingresar al sitio web, el país, estado, y código de área telefónico donde se ubican los servidores del proveedor de Internet del usuario, las páginas de este sitio web que los usuarios vieron durante su visita y cualquier término de búsqueda ingresado en este sitio web.
Este sitio web no está dirigido a personas menores de 13 años. eMetric no recopila a sabiendas información personal de niños (as) menores de 13 años a través de este sitio web. Se les solicita expresamente a los estudiantes menores de 13 años que no proporcionen ninguna información personal a través de este sitio web. Si se da cuenta que un niño o niña ha proporcionado información de carácter personal a través de este sitio web, por favor contacte a eMetric al (877) 829-7769.
La seguridad de su información personal es muy importante para nosotros. eMetric utiliza medios comercialmente aceptables para proteger cualquier información personal recabada a través de este sitio web. Sin embargo, a pesar de nuestros mejores esfuerzos, ningún método de transmisión por Internet es completamente seguro. No podemos garantizar su seguridad absoluta. eMetric no asume responsabilidad por el uso de su información de carácter personal por parte de terceros.
Este sitio web puede incluir enlaces a sitios web de terceros. Cuando usted se conecta a otro sitio fuera del dominio de emetric.net, esta Política de Privacidad ya no aplica. Cuando usted se conecta a otro sitio web, está sujeto a la política de privacidad del nuevo sitio web.
La Política de Privacidad para este sitio web puede ser revisada ocasionalmente, por medio de una publicación actualizada. Si se hacen cambios significativos a esta Política de Privacidad, estos cambios serán publicados en este sitio web para ponerle sobre aviso. Por lo tanto, debe revisar esta Política de Privacidad periódicamente. Estas revisiones tendrán vigencia a partir de su publicación y su uso continuo de este sitio después de la publicación de dichas revisiones indicará su aceptación de tales revisiones.
Si tiene alguna pregunta o inquietud sobre esta Política de Privacidad, por favor llámenos al (877) 829-7769. Si quiere que respondamos a su comentario o pregunta, por favor proporcione su información de contacto en su mensaje.