In online research, the main sources of inaccuracy in regards to timing come from peripheral devices, which affect both native applications (i.e. installed on a computer in the lab) and browser-based ones alike. Woods et al. (2015) give a good overview of the issues involved in online research, and contains a detailed section on timing.
In Gorilla, we use high resolution timers for accurate reaction times and frame counting for accurate stimuli presentation times - techniques that have only recently become available in all major browsers.
Click through sections on the menu to the left to learn all about timing in Gorilla and make sure to check out further resources for more information on timing in online research!
When conducting research online, rather than in the lab, it's important to understand how timing accuracy changes.
The schematic below shows the main differences:
Modern techniques refers to the
Performance.now() function common in all major browsers which provides microsecond timing (read more on Mozilla's
Performance.now() page). We do not have control
of the underlying operating system, so we cannot make the same OS-level timing calls that native software
can. While clearly there will always be some experiments that require extremely high fidelity timing, the
precision offered by Gorilla is appropriate for a wide range of research.
The latency between the cloud storage and local computer is irrelevant because Gorilla downloads all task information (i.e. stimuli) to the local machine in advance and collects local timestamps.
To talk through each element in the picture above:
See the further resources section for more information on timing in online research.
Within the last few years, more accurate techniques have become available:
performance.now() offers microsecond-precision timestamps. This was first introduced in Chrome in 2012, and iOS was the last to adopt it in 2015.
requestAnimationFrame() allows us to run logic every time the screen is about to be refreshed, and should be synchronised to the refresh rate. This was first introduced in Firefox in 2012, and Android was the last to adopt it in 2013.
The key issue with displaying stimuli is that we are fundamentally limited to the refresh rate of the display. Most modern displays refresh at 60Hz, which is every 16.67 milliseconds. This means that stimuli can only ever be displayed for multiples of 16.67 milliseconds - without specialist equipment, you cannot show a stimuli image for 40ms; you can either have 33.3ms (two frames) or 50ms (three frames).
This issue affects native programs and browser-based ones equally, as they are both ending up on the same display.
requestAnimationFrame() to count frames and ensure that a stimuli is shown
for as accurate a time as possible, by evaluating for each frame whether it would be more accurate to
display for another frame (because there is more than 8ms left to display) or to stop on this frame (because there is less than 8ms left to show).
Whenever a new screen is displayed in Gorilla, a timestamp is recorded using
When a response is recorded (e.g. a keypress or a mouse click), a second timestamp is taken, and the reaction time
is then the difference between the two.
Exactly how accurate the response is depends chiefly on the latency of the input device. The default USB polling rate in Windows is 120Hz, or every 8.3ms. Therefore, there could be up to an 8.3ms error on keyboard presses or mouse clicks for peripherals connected via USB.
Touchscreens have higher latency still. The fastest devices are now at around 20ms, and getting faster all the time.
We published our own paper on timing accuracy in Gorilla (Anwyl-Irvine et al., 2019) where we used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the “conflict network” effect in all these populations, demonstrating the platform’s capability to run reaction-time-sensitive experiments - the effect we replicated was under 100ms.
We've also published a paper that focuses on timing accuracies on various online platforms, browsers, and devices (Anwyl-Irvine et al., 2021).
UCL are using Gorilla for teaching first year undergraduate students. One of the first research methods lectures was on the IAT. 108 participants took part in an IAT that is included in the Starter Project: Examples Project folder. This IAT investigates whether participants have an implicit association between young people and good words, and old people and bad words.
The results of this test replicated previous findings that the participants do have such an implicit association.
Below are some highly recommended resources to help you learn more about timing in online research:
Hilbig (2015) presents the effect of lab- versus web-based research on reaction times.
Woods et al. (2015) wrote an excellent summary of the strength and weakness of online research, and the paper contains a detailed section on timing.
Our own paper, Anwyl-Irvine et al. (2019), discusses timing accuracy in Gorilla.
Our recent paper, Anwyl-Irvine et al. (2021), compares timing accuracies on various online platforms, browsers, and devices. This BeOnline talk "All about timing accuracy" presents the results from the pre-print and discusses the accuracy and consistency of visual presentation and recording of keyboard responses online. We also have a blog post summarising our timing performance article.
Katika utafiti wa mtandaoni, vyanzo kuu vya ukosefu wa usahihi kuhusiana na muda hutoka vifaa vya pembeni, ambayo huathiri maombi ya asili (yaani, Ni moja ya kompyuta ya maabara ya kompyuta, na ni moja ya browser-msingi. Woods et al. Kwa upande mwingine, kuna maelezo ya kina ya utafiti wa mtandaoni na sehemu ya kina kuhusu muda. Katika Gorilla, tunatumia vipima-wakati vya azimio la juu kwa nyakati sahihi za athari na kuhesabu fremu kwa nyakati sahihi za uwasilishaji wa kichocheo - mbinu ambazo zimepatikana hivi karibuni tu katika vivinjari vyote vikuu. Bonyeza kupitia sehemu kwenye orodha ya kushoto kujifunza yote kuhusu muda katika Gorilla na kuhakikisha kuangalia nje rasilimali zaidi kwa habari zaidi juu ya muda katika utafiti online! Wakati wa kufanya utafiti mtandaoni, badala ya katika maabara, ni muhimu kuelewa jinsi usahihi wa muda unabadilika. Mfano wa chini unaonyesha tofauti kuu: Mbinu za kisasa zinarejelea kazi ya Performance.now () ya kawaida katika vivinjari vyote vikuu ambayo hutoa wakati wa microsecond (soma zaidi kwenye ukurasa wa Mozilla wa Performance.now ()). Sisi si kuwa na udhibiti wa mfumo wa uendeshaji msingi, hivyo hatuwezi kufanya sawa OS-kiwango wakati wito kwamba programu asili inaweza. Ingawa ni wazi daima kutakuwa na baadhi ya majaribio ambayo zinahitaji sana high-uaminifu muda, usahihi zinazotolewa na Gorilla ni sahihi kwa ajili ya mbalimbali ya utafiti. Wakati wa kusubiri kati ya hifadhi ya wingu na kompyuta ya ndani haijalishi kwa sababu Gorilla hupakua habari zote za kazi (yaani, stimuli) kwa mashine ya ndani mapema na hukusanya timestamps ndani. Kwa kila kipengele katika picha hapo juu, angalia sehemu ya rasilimali zaidi kwa habari zaidi juu ya wakati katika utafiti wa mtandaoni. Katika miaka michache iliyopita, mbinu sahihi zaidi zimepatikana: performance.now() inatoa timestamps microsecond-usahihi. Hii ilitolewa kwa mara ya kwanza katika Chrome mnamo 2012, na iOS ilikuwa ya mwisho kuichukua mnamo 2015. requestAnimationFrame inaruhusu sisi kuendesha mantiki kila wakati skrini inakaribia kusasishwa, na inapaswa kusawazishwa na kiwango cha kusasisha. Ilitengenezwa kwa Firefox mnamo 2012 na Android ilikuwa ya mwisho kuitumia mnamo 2013. Suala muhimu na kuonyesha stimuli ni kwamba sisi ni kimsingi mdogo kwa kiwango cha refresh ya kuonyesha. Kwa mfano, skrini za kisasa huongeza kasi ya 60Hz kila milisekunde 16.67. Hii inamaanisha kwamba msukumo unaweza kuonyeshwa kwa mara nyingi za milisekunde 16.67 - bila vifaa maalum, huwezi kuonyesha picha ya msukumo kwa 40ms; unaweza kuwa na 33.3ms (mfumo mbili) au 50ms (mfumo tatu). Suala hili huathiri programu za asili na zile zinazotegemea kivinjari sawa, kwani zote mbili zinamalizika kwenye kuonyesha sawa. "Kutumia ""AnimationFrame"" ni njia ya kuhesabu picha na kuhakikisha kwamba stimuli inaonyeshwa kwa muda sahihi iwezekanavyo, kwa kutathmini kwa kila sura kama itakuwa sahihi zaidi kuonyesha kwa sura nyingine (kwa sababu kuna zaidi ya 8ms kushoto kuonyesha) au kuacha kwenye sura hii (kwa sababu kuna chini ya 8ms kushoto kuonyesha)." Wakati wowote skrini mpya ni kuonyeshwa katika Gorilla, timestamp ni kumbukumbu kwa kutumia Wakati majibu ni kumbukumbu (kwa mfano. Wakati wa kuchapisha (kuchapa kwa funguo au panya) ni wakati wa pili wa kuchapisha, na wakati wa majibu ni tofauti kati ya mbili. Ni jinsi gani sahihi majibu ni inategemea hasa juu ya latency ya kifaa cha pembejeo. Kiwango cha kawaida cha kupigia kura cha USB katika Windows ni 120Hz, au kila 8.3ms. Kwa hivyo, kunaweza kuwa na kosa la 8.3ms kwenye mibofyo ya kibodi au mibofyo ya panya kwa vifaa vya nje vilivyounganishwa kupitia USB. Vipimo vya kugusa vina latency ya juu zaidi bado. Vifaa vya haraka zaidi sasa ni karibu 20ms, na kupata haraka zaidi wakati wote. "Kama ilivyoelezwa katika makala ya ""Uhakika wa Wakati katika Gorilla"" (Anwyl-Irvin et al., 2019), tunafanya kazi ya ""Flanker"" iliyochukuliwa kutoka kwa kazi ya mtandao wa tahadhari (Rueda, Posner, & Rothbart, 2004)." Tulirudia athari ya "mitandao ya mgogoro" katika idadi hii yote, tukionyesha uwezo wa jukwaa la kuendesha majaribio ya unyeti wa wakati wa athari - athari tuliyoirudia ilikuwa chini ya 100ms. "Tumechapisha pia karatasi ambayo inazingatia usahihi wa wakati kwenye majukwaa mbalimbali ya mtandaoni, vivinjari, na vifaa (Anwyl-Irvine et al., 2021).""" UCL hutumia Gorilla kufundisha wanafunzi wa mwaka wa kwanza wa shahada ya kwanza. Moja ya masomo ya kwanza ya mbinu za utafiti ilikuwa juu ya IAT. "Wanafunzi wa shule ya msingi walihusika katika ""Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano wa Mfano.""" IAT hii inachunguza ikiwa washiriki wana uhusiano wa wazi kati ya vijana na maneno mazuri, na watu wazima na maneno mabaya. Matokeo ya mtihani huu yalirudia matokeo ya awali kwamba washiriki wana ushirika huo wa wazi. Hapa chini ni baadhi ya rasilimali sana ilipendekeza kukusaidia kujifunza zaidi kuhusu muda katika utafiti online: Hilbig (2015) inatoa athari ya maabara dhidi ya utafiti wa mtandao-msingi juu ya nyakati za athari. Woods et al. "Mchoro huu unaelezea kwa ufupi nguvu na udhaifu wa utafiti wa mtandaoni na ""Mtazamo wa Utafiti wa Mtandao"" na unaelezea kwa kina zaidi juu ya wakati wa utafiti." Karatasi yetu wenyewe, Anwyl-Irvine et al. (2019), inazungumzia usahihi wa wakati katika Gorilla. Karatasi yetu ya hivi karibuni, Anwyl-Irvine et al. (2021), inalinganisha usahihi wa wakati kwenye majukwaa mbalimbali ya mtandaoni, vivinjari, na vifaa. "Hii BeOnline hotuba ""Kila kuhusu timing usahihi"" inatoa matokeo kutoka preprint na inazungumzia usahihi na uthabiti wa maonyesho ya kuona na kurekodi majibu keyboard online." Pia tuna chapisho la blogi linalofupisha makala yetu ya utendaji wa wakati.