LinkUpBot
LinkUpBot is the name for LinkUp’s web crawler, an application designed to automatically collect job posting information from company career sites.
The LinkUpBot crawls company career sites and gathers links to all active, publicly available jobs, and parses the content for inclusion in the LinkUp job index. As a result, the LinkUp job index powers LinkUp’s job search engine and job market data solutions to provide:
- Premium search and job listings for job seekers
- Valuable exposure and candidate traffic for employer’s open and active job listings
- Insightful job market information for organizations looking to improve their businesses
How LinkUpBot interacts with your robots.txt file
LinkUpBot is built to obey robots.txt files. If your site specifically states ‘disallow’, the LinkUpBot will obey this directive.
The LinkUp User-Agent
LinkUp uses the User-agent: "Mozilla/5.0 (compatible; LinkUpBot/1.0; +https://www.linkup.com/linkup-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36" to identify itself.
IP Range for LinkUp
Requests from LinkUpBot will come from the IP address 24.144.65.32.
Requesting a Crawl Delay
LinkUp would like to crawl your site as fast as possible to ensure that there are no interruptions or issues for your human visitors. To slow the crawl down, use the following Crawl Delay Directive to only allow LinkUp to access your site once every 10 seconds:
User-agent: LinkUpBot
Crawl-delay: 10
If you have several pages to crawl, it may take a while to crawl your site. Therefore, we don't recommend adding a crawl delay larger than 30 seconds or LinkUp might not be able to finish the crawl of your site. For example, there are 86,400 seconds in a day which would allow LinkUp to access 8,640 pages in a single day.
Block All Bots Except LinkUp
Make sure you have the 'user-agent specific directive' above the `all bots` directive User-agent: *.
User-agent: LinkUpBot
Disallow:
User-agent: *
/Marketplace/
Blocking LinkUpBot from Crawling Your Site
To block LinkUpBot from crawling your site, add a slash after disallow. The slash denotes the root of the site. Adding this code will prevent LinkUpBot from being able to crawl your website.
User-agent: LinkUpBot
Disallow: /
LinkUpBot can be excluded from parts of your site, like subfolders. This syntax tells only the crawler not to crawl any pages that contain the URL string http://www.example.com/Marketplace/...
User-agent: LinkUpBot
Disallow: /Marketplace/
Please note that it may take up to 24 hours to discover changes made to your robots.txt.
About LinkUp
LinkUp is a leading job search engine empowering people with knowledge obtained from the largest, highest quality database of global job listings. Through proprietary technology, LinkUp indexes millions of job listings daily, direct from company career sites, providing unsurpassed value for job seekers, employers, and data buyers.
Have questions? You can contact us here.
