Skip to content →

2018/05/29: Reflections on failures and open notebook

My last post was more than twenty days ago. The reason for that was that I was just very busy and had to prioritise instance the pre-registration of the second experiment (schemaVR2, see here), which included a power analysis. In addition to starting the data collection for schemaVR2, I also worked on a syllabus for a summer teaching job and a poster for Pyschologie und Gehirn (Psychology and Brain), which is on the automaticity of word recognition (see here). Long story short, I did not have enough time.

In this entry, I want to reflect firstly on my experiences while running the two experiments and secondly I want to reflect on how I continue this open notebook because I think I need to adopt a different mode so that the time between posts does not extent over long periods such as the current one.

Now to running the experiments: Most of the problems with running the experiments had to do with me rushing things. In hindsight, I think it would have been better to take more time. For instance, I made last minute changes to schemaVR2 by adding a no memory/guess/familiar/remember judgement to the task, which definitely improved the design as it will allow me to investigate whether and how recollection and familiarity might be differentially involved in remembering the expected, neutral and unexpected locations of expected and unexpected objects. However while changing the code, I accidentally introduced an error, so that I had to cancel one appointment because the script were not running, which was very unfortunate. Another minor issue was that the instructions contained a small number of typos and incorrect labels. The latter is not important because none of the participants realised this. However, this as well as that some participants did not rate the correct stimuli could have been avoided if I had spent more time rigorously testing and checking the whole experiment. This is something I will definitely do in future experiments. In that regard, it is also advisable to do at least one test run of the experiment, where one notes down the responses by hand and then check whether the same data is saved. Due to a copy and paste error, I did not change the name of a variable in my C# script, which would have been catastrophic if I had not a) also saved the data to the Unity log and b) realised the mistake very early.

Rushing was also a problem while carrying out the experiments because sometimes I forgot to activate a certain script in Unity with the effect that some participants briefly saw some of the objects twice. I made a couple of change in that regard to make these kind of operator errors less likely.

Even though the problems/issues I describe above seem to be numerous, I still feel positive about the how I conducted my first immersive VR experiments. The results are interesting and are interpretable and the quality of the data is still very good. As I am using linear mixed effects, missing values are not as problematic anyway.

Currently, I am not really satisfied with how much and in what frequency I am posting though. As already noted, the last post is more than twenty days ago. However, I also need to acknowledge my limits. The last couple of weeks were extremely intense due to which I always has to prioritise working on other things (e.g. pre-registration or poster) over writing this entry. Unfortunately, I think that this will happen again, but in order to be less stressed out about this I decided to reserve the last two hours every Friday for writing entries. Of course, this does not mean that I am not going post R scripts etc. when I produce them.

Concrete resolutions for the future:

  • If possible have test runs for new experiments, where responses are written down by hand and compared to the data that is saved.
  • Switch from daily to weekly updates for the notebook. In general, I will try to spend the last two hours of my working day each Friday posting entries in the notebook.

Published in failure meta open lab diary project 1 schemaVR1 schemaVR2


Leave a Reply

Your email address will not be published. Required fields are marked *