Make google crawl a site when it is cloaked - google-crawlers

I host this website. And I order a domain for it. SO the site is hosten on somedomain.be/boring
I added a domain www.company.com
Now In the webmastertools of google if I want to add the property they wil not verify it despite the DNS record, TXT, TAG, and file.
How Can I fix this?

I changed the config to HTTP redirect. That way it can be crawled but then I changed it back to cloaking again.

Related

Google Load Balancer www to non-www redirects (2022 October Version)

I wanted to use Google Load balancer to do wwww to non-www redirects.
There is a Stack Overflow question asking the exactly same question. I followed the step but it did not work.
In particular, I selected "Prefix redirect" and clicked "Done" and "Update". However, later when I clicked it again, it became back to "Full path redirect."
My question: Do I need to add anything in the "path value"? If so, what should I add there? Thank you so much!
If you're creating a new HTTPS load balancer, You must and should Enable HTTP to HTTPS Redirect checkbox to create the redirect.
As per official doc, Path value field should be blank and Post enabled above, follow this Redirect Doc which guides you in redirecting the path perfectly.
Note : While adding redirect path configurations you need to save and then click done and update. As you are not saving This might be the reason you are getting a full path redirect option when you cross check.

Unable to add custom domain to github page

I am absolutely new to GitHub so this may come out as a silly question. I am trying to host my website using GitHub pages. The website got hosted properly, but when I added the custom domain, it stopped working and now I am getting a blank page. What I have done is, have created a subdomain in my AWS-hosted zones (subdomain.example.com) and pointed it to https://my_org.github.io/my_repo/
Then I added subdomain.example.com in github page custom domain and CNAME file in the repository.
The error I am getting is
DNS check unsuccessful
subdomain.example.com is improperly configured Domain's DNS record
could not be retrieved. For more information, see documentation
(InvalidDNSError).
I am not sure what I am missing exactly. Does anyone have any idea about this?
It might be that you need to verify your domain name through a TXT record:
https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/verifying-your-custom-domain-for-github-pages

Understanding Server/Client Routing: How Can Amazon(?) Be Redirecting My SPA ... Without a Redirect (or History Entry)?

NOTE: I'm providing details of my setup, but really this is a "how is this possible" question, not a "please debug my setup" question.
I have a "singe page application" (ie. an HTML file that uses the History API to simulate URLs). I'm serving this app on AWS S3, behind an AWS Cloudfront ... front.
I had successfully configured things so that if someone went to www.example.com/foo (let's pretend I own example.com), Cloudfront would serve an "error page" of my index.html. My index.html would then see the URL, and use its routing to show the user the correct page.
That all worked great ... until it didn't. Now for some reason when I go to www.example.com/foo, I get redirected to www.example.com. I'm trying to debug things, but what I can't understand is how I'm going from /foo to the main page.
When I look in the Network panel of my developer tools, I can see the request made to the original (/foo). Then I can see the chain of requests (for images, css files, etc.), and they all have a referrer of www.example.com/foo.
Then all of the sudden I see a request for React Developer tools (why it needs to make a request is beyond me) ... and it's from referrer www.example.com. After that I get one last image request from /foo, and then all subsequent requests come from www.example.com.
Can anyone explain how this could be working? I know that if a server returns a redirect (either type) that could change my URL ... but every request has a 200 status (ie. no server redirects).
I know Javascript could "push" a new URL to my browser ... but that would leave a history entry right? When I go "back" (either with my browser or history.back()) I go to the page before; I don't go "back" to /foo.
So somehow I'm not making a history entry, but I am switching my URL, and the URL I make requests from, and this all happens within milliseconds on page load ... without any redirects. How?
P.S. When I use my dev tools to add an beforeunload breakpoint, then try to navigate from example.com to example.com/foo I don't hit that break point (either for going to /foo, or when I'm "redirected" back to example.com).
When I check the box for any Load event, I do see some happen ... after my URL has already switched. In other words, I type example.com/foo, hit enter, and by the time any event fires I'm back on example.com. Whatever mechanism is doing the "redirection" here ... it doesn't trigger any load events.
I figured out my (AWS-specific) problem, thanks to a bit of Gatsby documentation. I'll include the details below in case it helps others, but I won't accept this answer, as I still don't understand how AWS did what it did (and I'd still welcome an answer for that).
What happened was that I had my Cloudfront "Origin Domain Name and Path" pointing to:
example.com.s3.amazonaws.com
However, as explained on https://www.gatsbyjs.com/docs/deploying-to-s3-cloudfront/:
There are two ways that you can connect CloudFront to an S3 origin. The most obvious way, which the AWS Console will suggest, is to type the bucket name in the Origin Domain Name field. This sets up an S3 origin, and allows you to configure CloudFront to use IAM to access your bucket. Unfortunately, it also makes it impossible to perform serverside (301/302) redirects, and it also means that directory indexes (having index.html be served when someone tries to access a directory) will only work in the root directory. You might not initially notice these issues, because Gatsby’s clientside JavaScript compensates for the latter and plugins such as gatsby-plugin-meta-redirect can compensate for the former. But just because you can’t see these issues, doesn’t mean they won’t affect search engines.
In order for all the features of your site to work correctly, you must instead use your S3 bucket’s Static Website Hosting Endpoint as the CloudFront origin. This does (sadly) mean that your bucket will have to be configured for public-read, because when CloudFront is using an S3 Static Website Hosting Endpoint address as the Origin, it’s incapable of authenticating via IAM.
Once I changed my Cloudfront "Origin Domain Name and Path" to the bucket's static hosting URL:
http://example.com.s3-website-us-west-1.amazonaws.com
Everything worked!
But again, I still don't understand how AWS did what it did when I mis-set my "Origin Domain Name and Path". It redirected me to my root domain, seemingly without either a redirect response OR a client-side redirect, and I'd love to hear how that was accomplished.

Wagtail internal link urls arent working properly

in my rich body text fields, when an internal page is used for a link, the url that gets attached to the a tag is "https//example.com/example", ie it's missing the colon and the link doesnt work. I get the error "https's server ip address could not be found".
any idea why it is doing this? thanks
Check the site record under Sites -> Settings. The hostname field should NOT contain https - i.e. it should be example.com, not https://example.com.
in case anyone else has this issue, it seems having more than one site in wagtail caused this. I had the default localhost site still on there, even after switching to a new production site configuration. deleting the localhost site fixed this.

How to resolve "Domain's DNS record could not be retrieved" in GitHub page creation process?

I am trying to create a GitHub page for a repository. But when I gave the custom domain name, it shows the following message "Domain's DNS record could not be retrieved. For more information"
As I am new to GitHub I am not getting the information what is documented in GitHub pages. Could anyone help me to resolve this problem?
If you've recently changed or removed your custom domain and can't access the new URL in your browser, you may need to clear your browser's cache to reach the new custom domain. For more information on clearing your cache, see your browser's help site.
In order to serve the Page, your DNS records must point to GitHub's server. To confirm that your custom domain points to GitHub's servers, use the dig command with your custom domain. The dig command shows you where your custom domain points. For example:
$ dig example.com +nostats +nocomments +nocmd
example.com. 3600 IN A 185.199.108.153
In the example above, example.com points to the IP address 185.199.108.153.
If you configured A records through your DNS provider, your A records must point your custom domain to the following IP addresses:
185.199.108.153
185.199.109.153
185.199.110.153
185.199.111.153
You may see a different IP address, since we serve Pages with a global Content Delivery Network. Use dig username.github.io to see the full resolution path. Note that DNS caching may cause a delay.
If you're using an A record that points to 192.30.252.153 or 192.30.252.154, you'll need to update your DNS settings for your site to be available over HTTPS or served with a Content Delivery Network. For more information, see "HTTPS errors."
If you're using an A record that points to 207.97.227.245 or 204.232.175.78, you'll need to update your DNS settings, as we no longer serve Pages directly from those servers.
Source: https://help.github.com/en/articles/troubleshooting-custom-domains