Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

Amateur mathematicians are using artificial intelligence chatbots to solve long-standing problems, in a move that has taken professionals by surprise. While the problems in question aren’t the most ...