Senior Web Crawling Engineer

A stealth, engineering-driven startup with significant funding is looking for a Senior Web Crawling Engineer to design and own the core crawling system that powers their next-generation search platform.

The team builds deeply technical systems in Rust, though strong experience in C++ or another low-level language plus an interest in working with Rust is also welcome.

What You’ll Work On

• Architect and implement large-scale web crawling infrastructure
• Manage crawling pipelines: scheduling, graph traversal, deduplication, and robustness
• Integrate crawled data with downstream indexing and storage systems
• Contribute to high-performance Rust systems used across the product

Requirements

Must-have

• Experience building large-scale web crawlers (millions of URLs or comparable scale)
• Strong background in systems programming (Rust, C++, or similar)
• Ability to design, build, and maintain complex distributed systems
• Solid understanding of HTTP, robots.txt, sitemaps, rate limiting, and crawling best practices

Nice-to-have

• Production Rust experience or a strong desire to work in Rust
• Familiarity with search, indexing, or storage systems
• Experience with low-latency or high-throughput distributed systems
• Startup-oriented, hands-on engineering mindset

About the Company

• Well-funded stealth startup (8-figure funding)
• Building a new search engine layer for AI and autonomous agents
• Small, highly technical engineering team
• Fully remote, with preferred overlap across European / Eastern / Pacific time zones
• Flexible arrangements (contract or EOR available)

How to Apply

Send your resume and a brief summary of previous crawling or systems work to webcrawling@rustjobs.dev