pointertovoid Posted March 7, 2011 Share Posted March 7, 2011 (edited) Hello dear readers!Ignoring nearly everything about websites, I'd like you to explain me why Google occults a website, or rather parts of it, in its search results.Raw elements:- Forum not listed, nor its "Sciences" section , but does list the "General discussions" section - Other search engines occult the forum as Google does- Said forum describes no drugs, explosives, misconducts... But it does contain my inventions (civilian uses) in the occulted sections- Other forums where I put a few inventions, and links to this Saposjoint, are listed by Google- Until recently, I didn't receive e-mails sent to myself containing an address of the Saposjoint website.Questions:- I browse from the semi-free country I happen to inhabit presently, called France, whose official policy is to censor some websites, under various excuses (explosives and so on).Could you please check Google's answers from your country?- Can the Saposjoint website be programmed (Php) in such a way that search engines occult some sections?Thank you so much!Marc Schaefer, aka Enthalpy, aka Pointertovoid Edited March 7, 2011 by Tripredacus Link to comment Share on other sites More sharing options...
Tripredacus Posted March 7, 2011 Share Posted March 7, 2011 I can find those sections with Google. Are you only using google.fr?I notice you only have 1 keyword tho, just your website name. Link to comment Share on other sites More sharing options...
pointertovoid Posted March 7, 2011 Author Share Posted March 7, 2011 (edited) Thanks for helping!I get the same results through google.fr, google.com, google.de, google.es used from France, and from other search engines.When searching through Google some text picked from threads in the occulted sections, I don't get results. Examples:- "tinnitus, it is obnoxious" or "technological state just now than Krypton" - But right now, Google finds "remotely operated like Predators and Reapers", which it did NOT on 4th of March (today is 7th) despite the page is from February.Hey, Msfn has real magic effects!- Google doesn't find any "Marc Schaefer, aka Enthalpy" on forum - though I put hundreds there.- The occulted discussions are not the ones where I contributed nor the ones containing my name. But the occulted sections are the ones describing my inventions.More observations:- BAUTforum.com crashed regularly as I attempted to put links there to forum. But other forums did not.- Now I receive emails from myself containing a link to forum, which was still impossible on 13th of February.I had to relay to myself the sent message in order to receive the second try. Same without the antivirus nor the firewall. I checked the message rules for Outlook Express as well.This improved as I began to experiment on it systematically. One other magic effect.And I always received emails containing links to other forums quite normally.I ignore how Php works... The computer hosting forum re-creates the web page each time a visitor asks for a page, as opposed to Html, isn't it?Would it then be technically possible that forum is programmed to detect the bots sent by search engines, and produces for them expurgated pages?Though this alone wouldn't explain the censored e-mails nor the crashes at other web sites. Edited March 7, 2011 by Tripredacus Link to comment Share on other sites More sharing options...
Tripredacus Posted March 7, 2011 Share Posted March 7, 2011 (edited) I ignore how Php works... The computer hosting forum re-creates the web page each time a visitor asks for a page, as opposed to Html, isn't it?Would it then be technically possible that forum is programmed to detect the bots sent by search engines, and produces for them expurgated pages?Though this alone wouldn't explain the censored e-mails nor the crashes at other web sites.It has nothing to do with PHP. Most forum software works that way where it only "creates" pages as needed, as most of the data is kept in the database.You can try using a proxy in the US to see if you get different google results that way.Hey, Msfn has real magic effects!Indeed. I have removed all links and name to your forum. Edited March 7, 2011 by Tripredacus Link to comment Share on other sites More sharing options...
pointertovoid Posted March 7, 2011 Author Share Posted March 7, 2011 I have removed all links and name to your forum.Why? They were useful to readers willing to check Google's behaviour.By the way, it's Sapo's forum, not mine. I'm a member there, no other relation. Link to comment Share on other sites More sharing options...
pointertovoid Posted March 7, 2011 Author Share Posted March 7, 2011 The name of the forum there is saposjoint.netIts section called Science gives no hit in Google search, at least from here.But its section called "General discussions" does. Link to comment Share on other sites More sharing options...
jaclaz Posted March 7, 2011 Share Posted March 7, 2011 It is perfectly possible that there is a "robots.txt" preventing indexing a part of the forum and not another.http://en.wikipedia.org/wiki/Robots_exclusion_standardYou should ask the actual Forum Administrators if such a policy is in effect and how it is set.jaclaz Link to comment Share on other sites More sharing options...
submix8c Posted March 8, 2011 Share Posted March 8, 2011 (edited) Interesting... I Google what you said "General Discussions" and when I try to go to it I get "Forbidden" in IE6 but works (can go to it) fine in FireFox.Riddle me that... Maybe that's why Google can't pick up on it?(the website+)/Forum/viewforum.php?f=76Fine in FF, but in IE6,"Access forbidden!You don't have permission to access the requested object. It is either read-protected or not readable by the server"Try it... (I think the forum has "bugs") Edited March 8, 2011 by submix8c Link to comment Share on other sites More sharing options...
pointertovoid Posted March 9, 2011 Author Share Posted March 9, 2011 Internet Explorer : o yes, I had forgotten that. The site doesn't accept it here neither. Its webmaster, who has a Linux background, explicitly asks to use Firefox. Maybe he got some disappointments with IE, and his opinion about M$ won't have given him patience to solve them? So he probably put something likeif navigator==IE then goto hellI suppose this isn't the reason why search engines occult the site, because- I see often their bots in the visitor's list of that site- All sections of the site reject IE, but some are indexed by the search engines, while others aren't listed- My intuition tells that the bots sent by the search engines don't rely on a browser - but I may be wrong-----Robots.txt (Ciao Jaclaz, grazie!) would be a very credible explanation. Thanks for the link at Wiki!I had already tried to ask the administrator, who avoided giving a clear explanation.Fortunately, his robots.txt is accessible from Firefox, I have it under my eyes.It contains, among others:Sitemap: coughcough/sitemap_index.xmlDisallow: /Do I understand properly that sitemap_index.xml then gives the exclusive list of what the bots shall index?The sitemap_index.xml is accessible and contains for instance<sitemapindex etc><sitemap><loc>CoughCough/sitemap/0.xml</loc><lastmod>2011-03-05T12:00:00+00:00</lastmod></sitemap>And the just mentioned 0.xml contains<url><loc>CoughCough/viewforum.php?f=66</loc><lastmod>2010-11-07T17:28:04+00:00</lastmod><changefreq>daily</changefreq><priority>1</priority></url>where viewforum.php?f=66 is the section "Technology" that I'm most interested in, which search engines don't list.What would bots be supposed to do when reading this?Thanks! Link to comment Share on other sites More sharing options...
jaclaz Posted March 9, 2011 Share Posted March 9, 2011 (edited) What would bots be supposed to do when reading this?Most probably (but not really my field of knowledge):Get to the "Sitemap" line.From it get to sitemap_index.xml.From it get to 0.xml.Read the line "viewforum.php?f=66" and LEAVE FORUM #66 alone since it is disallowed as well as the whole site....http://en.wikipedia.org/wiki/SitemapsBut it could also be the other way round, like that the Forum Admin meant to have that forum crawled, but that *something* prevents the robots to follow the chain and the result is involuntarily to "disallow" it specifically. The "Disallow /" should prevent ANY robot from crawling ANY page, from what I understand, no matter what the Sitemap says....jaclaz Edited March 9, 2011 by jaclaz Link to comment Share on other sites More sharing options...
Tripredacus Posted March 9, 2011 Share Posted March 9, 2011 I have removed all links and name to your forum.Why? They were useful to readers willing to check Google's behaviour.By the way, it's Sapo's forum, not mine. I'm a member there, no other relation.Having all those links, so you could see that google results because they were on MSFN is too close to SEO spamming behaviour which is against the forum rules. So I either remove the links or you could be banned for spamming. So I removed the links so you can get help with the issue. Besides, they weren't specifically necessary to help with the question. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now