The patent application regarding WebVeta was rejected in the patent office based in U.K. They considered it like a algorithm. In a certain way yes and in a certain way no.
The patent was about a methodology for near real-time indexing.
The standard way is background processes to crawl websites or sitemaps or rss periodically. IndexNow protocol allows submitting the URL of the website. IndexNow requires some SDK / development.
For WebVeta I wanted a way for detecting changes, even for static websites. If I can do something on a static website using normal technologies such as HTML, Javascript, CSS then the same thing can be done for any dynamic website and developers of dynamic website do NOT have to do any special development for real-time indexing.
For WebVeta search, people would include Javascript anyway. I thought what if I do a content hash after the page loads (after removing some non-necessary elements such as head etc...) and send a background request to the web server. Now the server verifies the hash and if different or new, does indexing.
The problem with the above approach is possibility of DDOS i.e random people would generate requests from various IP's and send various hashes. Because WebVeta has built-in domain request crawl rate, the actual websites would not be bombarded with requests.
For resolving this problem, a unique key can be provided for the owner of the website. Now the author makes updates and needs to view the page by providing special key in the URL. Slightly frustrating.
If the pages are generated programmatically, SDK approach would suffice with some simple HTTP request.
This is not a great concept, but yes, I suppose this can be considered as just an algorithm and definitely yes useful for WebVeta.
Anyway, am I encouraging my competition by writing in a public blog post? Probably, but remember healthy competition would always be a win-win situation. Haters, psychos, negative mindset hackers, spies, stalkers, shadows etc... with communist/socialist/extremist agendas are literal gravity.
-
Mr. Kanti Kalyan Arumilli
B.Tech, M.B.A
LinkedIn - Temporarily Blocked :(
https://www.youtube.com/@alighttechnologyandservicesltd
https://www.youtube.com/@kantikalyanarumilli
https://twitter.com/KantiKalyanA/
Founder & CEO, Lead Full-Stack .Net developer
ALight Technology And Services Limited
Phone / SMS / WhatsApp on the following 3 numbers:
+91-789-362-6688, +1-480-347-6849, +44-07718-273-964
+44-33-3303-1284 (Preferred number if calling from U.K, No WhatsApp)
No comments:
Post a Comment