Remote
Senior Web Crawling Engineer
Stealth AI Startup Remote - United States 🇺🇸
Stealth AI Startup A stealth, engineering-driven startup with significant funding is looking for a Senior Web Crawling Engineer to design and own the core crawling system that powers their next-generation search platform.
The team builds deeply technical systems in Rust, though strong experience in C++ or another low-level language plus an interest in working with Rust is also welcome.
What You’ll Work On
- Architect and implement large-scale web crawling infrastructure
- Manage crawling pipelines: scheduling, graph traversal, deduplication, and robustness
- Integrate crawled data with downstream indexing and storage systems
- Contribute to high-performance Rust systems used across the product
Requirements
Must-have
- Experience building large-scale web crawlers (millions of URLs or comparable scale)
- Strong background in systems programming (Rust, C++, or similar)
- Ability to design, build, and maintain complex distributed systems
- Solid understanding of HTTP, robots.txt, sitemaps, rate limiting, and crawling best practices
Nice-to-have
- Production Rust experience or a strong desire to work in Rust
- Familiarity with search, indexing, or storage systems
- Experience with low-latency or high-throughput distributed systems
- Startup-oriented, hands-on engineering mindset
About the Company
- Well-funded stealth startup (8-figure funding)
- Building a new search engine layer for AI and autonomous agents
- Small, highly technical engineering team
- Fully remote, with preferred overlap across European / Eastern / Pacific time zones
- Flexible arrangements (contract or EOR available)
How to Apply
Send your resume and a brief summary of previous crawling or systems work to webcrawling@rustjobs.dev
Application process
- Your application goes directly to the Rust Jobs team. We manage hiring for this role on behalf of the company.
- Every CV is read by a human, not a filter. We assess it against the requirements before anything else happens.
- If your application is a strong fit, we'll reach out by email to introduce ourselves and walk you through the next steps with the company.
Application volume is high, so we can't reply to every candidate individually. If you haven't heard from us within 5 working days, please assume we're not moving forward on this occasion. Join our talent network to be considered for future roles.
Looking for more roles like this?
Join our talent network and get matched with similar opportunities from top companies.