Hide generated articles from search and index
Question
> The articles generated by the configurator appear in the Shopify search, on Google and are crawled by bots (e.g. Pinterest). How can I hide them?
Answer
K3 automatically adds a tag to each generated article: k3-generated
You can use this tag to remove items from the store search, make them invisible to search engines, exclude them from collections and prevent bot access to the K3 app URL.
For K3 to be able to add an item to the shopping cart, it must be published and listed in Shopify. A completely unpublished item cannot be added to the shopping cart. The following measures ensure that the articles are still invisible to end customers and search engines.
Overview of the measures
| Measure | What it does |
|---|---|
| 1. Filter store search | End customers no longer find the articles via the search in the store |
| 2. prevent search engine index | Google, Bing, Pinterest & Co. do not include the product pages in their index |
| 3. Exclude from collections | The articles do not appear in collection pages and navigation |
| 4. Block bot crawling of the app URL | Crawlers no longer call the /apps/k3/ route |
1. filter articles from the store search
Purpose: End customers who use the store search do not see K3-generated articles in the results.
Option A - Search & Discovery App (recommended)
The Shopify app Search & Discovery allows you to exclude products from search results based on tags - including from the Search API and Predictive Search. Set up a rule there that hides products with the k3-generated tag.
Option B - Liquid filter in the theme
Alternatively, you can filter the search results in the theme template:
{% unless product.tags contains 'k3-generated' %}
<!-- Produkt anzeigen -->
{% endunless %}
The Liquid filter only hides the articles in the rendered view. The Shopify search API (/search.json, Predictive Search) continues to deliver the hits. If you want to remove the items completely from the search, use Option A.
2. prevent search engine indexing (noindex)
Purpose: Search engines such as Google, Bing and Pinterest do not include the product pages in their search index. This means that the articles do not appear in external search results.
In the theme layout (theme.liquid) in the <head> area:
{% if product.tags contains 'k3-generated' %}
<meta name="robots" content="noindex, nofollow">
{% endif %}
Shopify automatically includes all published products in the sitemap - this cannot be prevented. However, this is not a problem: If a crawler finds the page via the sitemap and sees the noindex tag there, the page will not be included in the index. The sitemap leads the crawler to the page, the noindex prevents indexing.
3. exclude from collections
Purpose: The generated items do not appear on collection pages and therefore cannot be found via the store navigation.
Option A - Automatic collection with condition (recommended)
In the Shopify admin under Collection → Set conditions:
> Tag does not contain k3-generated
Then these items will not even be added to the collection.
Option B - Liquid filter in the collection template
{% for product in collection.products %}
{% unless product.tags contains 'k3-generated' %}
<!-- Produkt anzeigen -->
{% endunless %}
{% endfor %}
4. block bot crawling of the app URL (robots.txt)
Purpose: Search engine bots and other crawlers (e.g. Pinterest, GPTBot) no longer call the K3 app route /apps/k3/ directly.
When a crawler calls this URL, K3 creates a new item in the store on the server side - just like a real user who places a configuration in the shopping cart. This means that every bot call to the URL can create a new phantom item in the store. The entry in robots.txt instructs rule-compliant crawlers to ignore this route.
Shopify allows you to customize the robots.txt via a theme template. Create the file templates/robots.txt.liquid in your theme with the following content:
The template must not be indented. Each space in the template is transferred directly to the robots.txt output - leading spaces before Disallow: rules invalidate the robots.txt.
{%- for group in robots.default_groups %}
{{- group.user_agent -}}
{%- for rule in group.rules %}
{{- rule -}}
{%- endfor %}
Disallow: /apps/k3/
{% endfor -%}
The Shopify standard contains a comment block at the beginning of robots.txt, which refers bot operators to the Shopify terms of use (no automated checkout without human confirmation). This block cannot be accessed via Liquid and is omitted when using a custom template. If you want to keep it, you can copy it manually to the beginning of the template:
# we use Shopify as our ecommerce platform
# ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
# ┃ Robots & Agent policy ┃
# ┣━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
# ┃ Checkouts are for humans. ┃
# ┃ * Automated scraping, "buy-for-me" agents, or any end-to-end flow ┃
# ┃ that completes payment without a final human review step is not ┃
# ┃ permitted. ┃
# ┃ * Legitimate integrators must use the official Checkout Kit: ┃
# ┃ https://www.shopify.com/checkout-kit ┃
# ┃ ┃
# ┃ Terms of Service: https://www.shopify.com/legal/terms ┃
# ┃ Contact: bots@shopify.com ┃
# ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
The robots.txt is only observed by rule-compliant crawlers (e.g. Googlebot, Bingbot, Pinterest crawler). Link previews (WhatsApp, Slack, iMessage) and some marketing bots ignore robots.txt. Therefore, this measure is a useful supplement to the other measures, but not a sole protection.
Clean up: Delete old generated articles
The measures mentioned above prevent generated articles from being visible - but the articles themselves remain in Shopify. To keep the store clean, you can regularly delete old K3-generated articles that are no longer needed.
Manually: In the Shopify admin under Products, filter for the tag k3-generated and delete older entries without an associated order.
Automated: With Shopify Flow you can set up a workflow that automatically deletes K3-generated products after a defined time (e.g. 30 days) if no active order is linked to them.
Further information
- Shopify-Anbindung einrichten - Shopify integration basics