The Café Wall works by feeding our brain too much contrasting information at once. Our visual systems are not very precise when perceiving strong contrasts in color, and the strong black-and-white pattern with mortar in the middle on the Café Wall immediately overpowers the brain.
When we look at Cafe Wall we don’t notice the mortar as being separate from the alternating black and white tiles; we instead assume the mortar is a part of whichever tiles are closest to our field of vision. Since we think the mortar and the tile it is closest to are one object, our brains must then assume that the tile containing the mortar must be larger than the tile not containing the mortar. But then when we look at Café Wall, we see that all the tiles contain mortar! How can one object be bigger than another if they are all the same size and containing the same edging of mortar? They can’t.