AI Got It Wrong - Comprehension

We evaluate the ability of different AI systems to comprehend complex narratives. We use a modified version of the nursery rhyme "Jack and Jill," where they ascend a hill and return on their own two feet, unlike the original version where they tumble "head over heels." The story begins with Jack and Jill having 10 apples. However, it introduces additional details about the distribution of apples between them. Notably, Jill also has oranges, and at some point, Jack is said to pick up two shrivelled apples, which ultimately leads to the correct answer. Some AI systems incorrectly interpret the scenario. Most provide insight into their reasoning processes, which can help readers arrive at the correct conclusion by analysing these explanations. An intriguing observation from this exercise is that semantically confusing information can lead AI systems to produce incorrect outputs. For content creators, particularly in corporate environments where AI may be used to summar...