Perplexity is said to be allowing its AI to break the fundamental rule of the Internet

Perplexity wants to change the way we use the Internet, however AI research startup backed by Jeff Bezos You may be breaking their rules to do so. The company appears to be ignoring a widely accepted web standard, the Bot Exclusion Protocol, to delete parts of the web that operators don’t want accessed by bots, according to a report by Developer Rob Knight This week it was Confirmed by Wired.

Perplexity summarizes articles on the web, and claims to provide “reliable answers” with “no need to click on different links,” as stated in Blog post. In order to do this, Wired and Knight found that Perplexity ignores code (robots.txt files) intentionally written to block web crawlers. The two reports found that Perplexity uses an unlisted IP address to navigate around these robots.txt files and scrape websites anyway. Wired claims that its website blocked Perplexity’s web crawler earlier in 2024, but the AI ​​search engine was still able to summarize its articles in detail.

Despite this, Perplexity claims that it respects the bot exclusion protocol in documentation On its website. Aravind Srinivas, CEO of Perplexity, told Wired that reporters had a “deep and fundamental misunderstanding of how Perplexity and the Internet work,” but did not directly question the findings. Gizmodo has reached out to Perplexity to request a more detailed response and will update the article if we hear back.

Separately, Perplexity is currently facing legal threats for breaking some other Internet rule: copyright infringement. As reported in Forbes magazine He threatened to take legal action against Al-Hira This week, after the artificial intelligence startup was accused of… Tear up Forbes reports Without correct attribution. Forbes magazine did the original reporting on Former Google CEO Eric Schmidt’s AI-powered drone projectAnd Perplexity creates AI-generated articles, audio clips, and videos using Forbes text and images. Forbes’ executive editor lobbed the word Perplexity at X earlier this month.

Although Perplexity’s product is useful, it redirects your Internet traffic. Google also indexes web pages and provides short summaries with AI technology, but directs traffic directly towards the web pages where the information comes from. Perplexity effectively writes detailed articles with AI, making it so users don’t click through to websites, breaking the digital media business model.

OpenAI has been formed Partnership with media companies To address this problem, pay them upfront for the license content, and the confusion is It is reportedly working on similar content partnerships, but instead of paying a fixed fee for content like OpenAI, Perplexity aimed to share the revenue. But those partnerships don’t exist yet, so for now Perplexity appears to be jumping the paywall and scraping websites for all the information it needs to power its AI answers.