A company with fourteen services and a map of the country it claims to cover is three afternoons away from owning a website ten times its current size. This is about what happens to such a site between publishing and appearing in results, and how the Indexing Hub helps — within limits worth stating plainly.
The arithmetic arrives quietly. Someone points out that nobody searching from Gävle sees a page mentioning Gävle. The obvious fix is a page for the service in that place, which works, so it is repeated. Fourteen services across twenty-one counties is 294 pages, and the English layer, added for the foreign firms setting up operations here, makes it 588.
None of that is written down as a decision. It happens as a sequence of small sensible edits, each defensible on its own, and the site that results is one a search engine has to be persuaded to look at. The mechanics discussed below run through the indexing tools in the Semalt panel, where submission, sitemap parsing and the per-address record sit together.
Multiplication is how a twenty-page site becomes a six-hundred-page one
Grids of this shape are not inherently wrong. A national supplier genuinely does deliver different things in different places, and a reader in Skellefteå genuinely does want to know whether anybody will come out. The problem is that the grid is generated from a template while the differences between cells are not.
Twelve of the fourteen services are identical everywhere. Six of the counties have no depot and no staff. So roughly two hundred of the 294 pages differ from each other only in the place name and a line of introductory text, and there is no version of a crawler that has been designed to enjoy that.
- The cells are not equal. A service you deliver from a depot forty minutes away is a different page from one you would subcontract. Only the first has anything specific in it.
- The second language doubles the weakest half. Translating a thin page produces a thin page in another language, and the reader it was translated for did not want a county page in the first place.
- Nothing is ever removed. Grids grow when a service is added and never shrink when one is dropped, so old cells linger without links pointing at them.
| Cell type | Roughly how many | Has anything specific to say | Verdict |
|---|---|---|---|
| Service with a depot in the county | 30 | Address, hours, staff, local work | Build properly |
| Service delivered from a neighbouring depot | 60 | A delivery day and a travel time | Section on a region page |
| Service subcontracted locally | 90 | Nothing the customer would value | Leave unbuilt |
| Service offered on request only | 114 | A phone number and an apology | Leave unbuilt |
| Any of the above, English | 294 | Written for a reader who wants national scope | Translate services only |
Published, found, fetched, kept — and none of them implies the next
Between a page going live and a page appearing in results there are four separate events, each with its own failure mode. A page is published. Its address becomes known, usually through a link or a sitemap entry. A bot fetches it. Something decides whether it is worth keeping in the index at all.
Most confusion about indexing comes from treating these as one motion. They are not, and the gaps between them are where a regional grid loses most of its pages. An address can be known for months without being fetched. It can be fetched and then not kept.
| Stage | What has happened | Typical failure on a regional grid |
|---|---|---|
| Published | The page exists and returns 200 | Reachable only through a dropdown, so nothing links to it |
| Known | The address is on a list somewhere | In no sitemap, and orphaned in the navigation |
| Fetched | A bot has requested it | Queued behind two hundred near-identical siblings |
| Kept | It has been judged worth storing | Judged a duplicate of the page it was templated from |
| Shown | It appears for a query | Kept, but never the best answer for anything |
Crawl budget is what your site has earned, not what it has been given
The phrase invites a picture of a quota handed out per domain, which is misleading. What actually happens is closer to a revisiting habit: how often a crawler comes back, and how deep it goes when it does, follows from how much of what it found last time turned out to be worth having.
A site of twenty solid pages that change occasionally gets fetched thoroughly. A site of six hundred pages where four hundred are template variants gets sampled, and the sample is not chosen by you. This is the concrete cost of the grid: it does not merely fail to help the weak pages, it slows down the strong ones by putting them in a longer queue.
Filter and sort parameters
Every combination is a distinct address unless something says otherwise, and a grid multiplies them again.
- Canonical on the clean version
- Keep parameter links out of navigation
Chains of redirects from old structures
Each hop is a fetch, and a grid rebuilt twice leaves two generations of addresses behind it.
- Point old addresses straight at the final one
- Retire dead cells rather than redirecting them
Pages that genuinely differ
A regional page with its own address, hours, staff and two local cases reads as a distinct document.
- Local specifics above the fold
- Linked from the national service page
A shallow, honest structure
Three clicks from the front page to any page you care about, and nothing important behind a dropdown.
- Real links, not scripted menus
- One index page per region group
The uncomfortable conclusion for the grid-building company is that deleting pages is a crawl-budget technique, and usually the most effective one available. Two hundred cells removed is two hundred fetches that go somewhere else.
A thousand addresses a day, ten thousand in one submission
The Indexing Hub exists for the case where you know about pages a crawler has not reached yet. The URL tracker carries a daily budget of 1,000 addresses per account, and a bulk submission accepts up to 10,000 addresses in a single batch, which are two different numbers doing two different jobs: the batch is how much you can hand over at once, the daily budget is how quickly it drains.
URL tracker and bulk submission
For the moment a batch of regional pages goes live and none of them is linked from anywhere yet.
- Submission through IndexNow. Addresses are pushed to the IndexNow API, which serves GoogleBot and BingBot, instead of waiting for a crawler to arrive on its own schedule.
- Ten thousand per batch. A whole grid can be handed over in one go; the daily ceiling then governs how long it takes to work through.
- Live counters. Submitted, discovered and failed are tracked as running totals, so a systematic problem shows up as a shape rather than as one bad row.
For our 588-page site the ceiling is not the constraint anybody expected. The whole estate fits inside a single day's budget with room to spare, which means the interesting question is not capacity but sequence: which addresses go first, and which ones you would rather a crawler did not spend the visit on.
Three levels deep, a thousand files, two jobs at a time
A sitemap is a statement about which addresses you consider part of the site. Treating it as a dump of everything the content system can produce wastes the one clean signal you have about your own structure.
Sitemap submission and parsing
For estates where the map is generated by the content system and nobody has read it in a year.
- Upload or point at a URL. Either a file you supply or the address of a live sitemap, which matters when the map is regenerated nightly.
- Recursive parsing, three levels. Index files pointing at index files are followed down to three levels, which covers the way most content systems split large estates.
- Two jobs in parallel, twenty queued. Enough for a scheduled reprocessing of a large estate without the queue becoming the bottleneck.
Parsing the map is also the cheapest audit of the grid available. Reading back what the file actually contains routinely surfaces cells nobody remembers commissioning, retired services still listed for eleven counties, and both language versions of a page that only ever needed one.
A timestamp, a status code and an error detail per address
Each address in the tracker carries its own crawl record: whether a bot has visited, when, what came back, and what the error was if there was one. This turns arguments about indexing into a lookup, which is a larger improvement than it sounds.
The recurring conversation it ends is the one where a page is declared invisible. Either a bot came and got a 200, in which case the problem is the page rather than its discoverability, or a bot came and got something else, in which case the fix is technical and specific, or no bot has come at all, in which case the page is orphaned and no amount of rewriting will help.
| What the record shows | What it means | What to do next |
|---|---|---|
| No visit recorded | The address is not reachable by links | Link it from the national page, resubmit |
| Visit, 200, still not in results | Fetched and judged not worth keeping | Give the cell content it does not share with siblings |
| Visit, 3xx chain | Budget spent on hops | Point the old address at the final destination |
| Visit, 4xx | The sitemap is listing something that is gone | Remove from the map before resubmitting anything |
| Visit, 5xx at a particular hour | The server is failing under a crawl burst | A hosting question, not a search one |
Prune first, submit second, and let the second language wait
The order matters more than any individual step. Submitting a grid before pruning it means spending discovery effort on pages you will remove in six weeks, and teaching a crawler that your site is mostly filler at the precise moment you were hoping to teach it otherwise.
The English layer belongs at the end of the sequence for a reason that is not technical. The reader it exists for — the firm opening a plant or a hub here, hiring, and buying from a dozen suppliers in a hurry — is not looking for a county page. They are looking for a supplier that operates in the country, in English, at a scale that will not collapse. Almost every English regional cell is a page built for nobody, and the correct handling of a page built for nobody is not to index it faster.
Worth the fetches, once pruned
The domestic reader is looking for proximity, so a cell backed by a depot answers a question the national page cannot.
- Keep cells with staff or stock behind them
- Submit these before anything else
Mostly pages built for nobody
The inbound reader wants national coverage in English; a county cell reads as a supplier who cannot reach them.
- Translate services, not regions
- Leave the cells unbuilt rather than unindexed
- Delete the cells with nothing in them. A service you subcontract in a county you visit twice a year does not need an address of its own.
- Fix what the survivors point at. Every surviving cell wants a link from the national service page and from the region index, not just from a dropdown.
- Rebuild the map, then read it. Parse it back through the Hub and confirm the count matches the number of pages you meant to keep.
- Submit in priority order. National pages, then real regional pages, then anything else that survived the first step.
Two campaign levels sit behind this work when it is more than one person can carry: AutoSEO at 149 USD per month per domain automates keyword discovery, link building and on-site suggestions, while FullSEO at 500 USD per month adds manual keyword selection with an automatic fallback, placements against a domain-rating target, a human review mode for on-site changes, and a team of specialists, developers and writers. Placements draw on a partner network of more than 230,000 sites, and first measurable movement usually takes four to eight weeks. The structural side of the grid is covered further under our services.
Questions that come up once the grid exists
We have already built two hundred regional pages. Delete them, or improve them?
Improve the ones you could defend in a sales meeting and delete the rest. The practical test is whether someone at the company could add two sentences to the page that are true only of that county. If nobody can, the page will not become distinct through editing, because there is nothing distinct to say.
How quickly should a submitted address appear in results?
There is no promised interval, and this is the point of the warning above. Submission affects discovery, not the decision that follows it. A well-linked page on an established site is often fetched within days; a thin page on a large grid may be fetched and then never kept at all.
Is 1,000 URLs a day enough for a catalogue of fifty thousand products?
Not for a full pass, which would take about fifty days. Catalogues of that size are not maintained by submitting everything anyway. You submit what changed, what is new and what matters commercially, and let the crawler handle the steady state through sitemaps and internal links.
Should the Swedish and English versions of a page be submitted separately?
Yes, they are separate addresses. The more useful question is whether the English version should exist. For service and capability pages it clearly should; for county-level cells it usually should not, and not submitting a page you should not have built is the cheaper of the two fixes.
Our sitemap lists eight hundred addresses but only four hundred pages are real. Does that matter?
It does. The map is read as a claim about what belongs to the site, and four hundred dead entries make the whole file less useful as a signal. Parse it, compare it against what you meant to publish, and cut it back before submitting anything else.
The fastest indexing decision is usually a deletion
Everything above points the same way. The Hub is genuinely useful for the case it addresses — new addresses that nothing links to yet, a rebuilt structure, a map that has drifted from reality — and it is honest about being a discovery instrument rather than a guarantee. What it cannot do is make a page worth keeping.
For a company that is national on paper and regional in practice, the grid is a way of writing down an ambition rather than a description of the business, and search engines read it as what it is. The pages worth keeping are the ones where a customer in that county would recognise something true about you; the rest cost fetches that the good pages needed. Related pieces on structure and measurement are collected in the English blog, and the practical starting point is a parse of your own sitemap to see how many addresses you are actually asking anyone to read: open the panel and run the first job.
