The site has enough capability to support content access by machine and this is permitted as long as usual rules are followed.
We recommend to start crawling from the big category http://ultrastudio.org/en/Category:Articles or from the sitemap http://ultrastudio.org/rss.xml?limit=100000. Both contain links to all production articles. http://ultrastudio.org/rss.xml may be useful to check new articles only.
Robots must not enter search, history or editing special pages (best is to avoid all pages starting with the prefix Special: in the url). Properly written robots should follow these rules automatically as they are defined in our robots.txt.
It is strictly forbidden for any robot to index or follow any page that is tagged with the standard directive <META NAME="ROBOTS" CONTENT="noindex,nofollow">. Such pages contain the active Java applets rather than links to launch them. Respecting the meta directive will make sure that user will always land on applet-free page where she can launch applets when required and if required. Also, these sites contain the duplicate content, apart from the applet launch tags.
If your bot already has some cached copy, we strongly recommend to use IF-MODIFIED-SINCE header that we fully support. This will boost a performance of your bot and will also save our bandwidth. This header must be represented as:
If-Modified-Since: Sat, 29 Oct 1994 19:45:30 GMT
For Java, this can be parsed and generated by the format string "EEE, d MMM yyyy HH:mm:ss z".
Our server will check the database is the requested topic has been edited after the indicated date. If not, it will return 304 (not modified) response without message-body.
To maintain the site operable, we may need to block bots that do not respect meta tags and robots.txt, especially if they create excessive load.
Exceeding the defined visit (about one page per second for several minutes) or traffic (over 50 Mb per hour) limits will trigger Ultrastudio.org denial of service attack prevention filter, blocking all access for the configured time. If this repeats, we may block by hand permanently. We may also block the bot on our discretion if clearly suspect DOS-like attack rather than normal usage.
New articles (RSS)