Setting up a new application

To use Algolia search or an Algolia chatbot, configure the Algolia dashboard and then wire the keys into the project.

Algolia’s own docs cover edge cases in more depth. Use them when a setting is unclear: Algolia documentation.

Getting started

In the Algolia dashboard, an application is the full setup: billing plan, crawlers, indices, AI assistant settings, and related resources.

3di has an account with demo and template applications that you can refer to. For a customer project, create a new application under that customer’s ownership and billing where required.

Creating an application

From the Algolia dashboard top bar, open the Application switcher.
Select Create Application.
Choose the plan, pricing, and region, then confirm.

To make your application work with the site, you need the API keys.

Application ID

To see the Application ID (like FEQQQ5LJP8), select the application name in the top bar.

Getting the API keys

Go to Settings > API Keys.
Copy:

Key	Use
Search-Only API Key	Frontend / browser — public, read-only
Admin API Key	Backend, crawler, and dashboard automation — keep secret

danger

Never commit the Admin API Key to the repo or expose it in client-side code. Use CI secrets or a secure vault for crawler and server-side jobs.

Setting up an index

An index holds search records for the chatbot. You can upload records or fill the index with a crawler. This guide assumes a crawler.

Go to Data sources > Indices.
Select Create Index.
Name the index so it matches the name you will use in the project config later.
3di chatbot indexes usually include for LLM in the name.
Open the index, then select Configuration.
Under searchable attributes, set values such as title, description, keywords, and url, or leave the default to search all attributes.

You can tune other index settings later if the chatbot results need it.

After records exist, browse them under Data sources > Indices > Browse.

Setting up a crawler

Go to Data sources > Crawlers.
Select Add new crawler.
Complete the wizard (start URL, schedule, and related options).

You can refine crawler behaviour after the first run. Use an LLM against the Algolia crawler docs when you experiment with content types and extractors.

Customising the crawler

Go to Data sources > Crawlers, then open your crawler.
Select Configuration.
Under initial set-up, set the schedule (for example twice a week) and the start URL.

You usually need to edit the crawler setup in the Editor (JavaScript) as well.

Typical fields:

Field	Meaning
`appId`	Algolia Application ID
`apiKey`	Admin API Key (not the search-only key)
`indexName`	Must match the index name used in the project
`schedule`	When the crawler runs (for example on Monday and Thursday)
`sitemaps`	Valid sitemap URL for the published site
`startUrls`	Starting URL (usually the home page)
`recordExtractor`	Custom extraction for docs body vs chrome
`exclusionPatterns`	Paths or file types to skip

Keep the crawl small

exclusionPatterns matter more than they look. A crawl that is too large can fail or burn through quota. Exclude binaries, archives, and sections that should not appear in search.

If the published site sits behind HTTP authentication, configure crawler login according to Algolia’s auth docs for your host. Prefer secrets over hard-coded credentials in the editor.

Template crawler example

The sample below is a working starting point for this Docusaurus template. It builds two actions:

A pages index for general site records
A Markdown-for-LLM index that extracts topic body text and drops TOC and chrome

Replace YOUR_APP_ID, YOUR_ADMIN_API_KEY, the site URLs, and the index names before you save it in the Algolia Editor.

Example crawler configuration

new Crawler({
  appId: "YOUR_APP_ID",
  indexPrefix: "",
  rateLimit: 8,
  maxUrls: 500,
  schedule: "on monday, thursday",
  startUrls: ["https://YOUR_SITE_HOST/"],
  sitemaps: ["https://YOUR_SITE_HOST/sitemap.xml"],
  saveBackup: false,
  ignoreQueryParams: ["source", "utm_*"],
  renderJavaScript: true,
  actions: [
    {
      indexName: "YOUR_PAGES_INDEX_NAME",
      pathsToMatch: ["https://YOUR_SITE_HOST/**"],
      recordExtractor: ({ url, $, helpers, contentLength, fileType }) => {
        // Drop TOC / chrome before default page extract
        $(
          [
            ".theme-doc-toc-desktop",
            ".theme-doc-toc-mobile",
            ".table-of-contents",
            "nav.pagination-nav",
            ".theme-doc-footer",
            ".breadcrumbs",
            ".hash-link",
            ".no-print",
            "script",
            "style",
          ].join(", "),
        ).remove();
        $("p").each((_, el) => {
          if (/^On this page:?$/i.test($(el).text().trim())) {
            $(el).remove();
          }
        });

        return helpers.page({ $, url, contentLength, fileType });
      },
    },
    {
      indexName: "Markdown for LLM",
      pathsToMatch: ["https://YOUR_SITE_HOST/docs/**"],
      recordExtractor: ({ $, url, helpers }) => {
        // Never target `main` — it includes the right-hand "On this page" column.
        $(
          [
            ".theme-doc-toc-desktop",
            ".theme-doc-toc-mobile",
            ".table-of-contents",
            "nav.pagination-nav",
            ".theme-doc-footer",
            ".breadcrumbs",
            ".theme-doc-version-badge",
            ".theme-doc-version-banner",
            ".hash-link",
            ".no-print",
            "script",
            "style",
          ].join(", "),
        ).remove();
        $("p").each((_, el) => {
          if (/^On this page:?$/i.test($(el).text().trim())) {
            $(el).remove();
          }
        });

        // Prefer the doc markdown node (actual topic body)
        let container = null;
        if ($(".theme-doc-markdown").length) {
          container = ".theme-doc-markdown";
        } else if ($("article .markdown").length) {
          container = "article .markdown";
        } else if ($("article").length) {
          container = "article";
        }

        if (!container) {
          return [];
        }

        const text = helpers.markdown(container);
        if (!text || text.trim() === "") {
          return [];
        }

        // Reject TOC-only leftovers
        const trimmed = text.trim();
        if (
          /^On this page:/i.test(trimmed) &&
          trimmed.length < 1200 &&
          (trimmed.match(/#/g) || []).length >= 2
        ) {
          return [];
        }

        const title = $("head > title").text().trim();
        const h1 = $(`${container} h1`).first().text().trim();

        return helpers.splitTextIntoRecords({
          text,
          baseRecord: {
            url: url.href,
            title: title || h1,
            heading: h1,
            lang: $("html").attr("lang") || "en",
          },
          maxRecordBytes: 5000,
          orderingAttributeName: "part",
        });
      },
    },
  ],
  initialIndexSettings: {
    YOUR_PAGES_INDEX_NAME: {
      distinct: true,
      attributeForDistinct: "url",
      searchableAttributes: [
        "unordered(keywords)",
        "unordered(title)",
        "unordered(description)",
        "url",
      ],
      customRanking: ["asc(depth)"],
      attributesForFaceting: ["lang"],
    },
    "Markdown for LLM": {
      attributeForDistinct: "url",
      distinct: 1,
      attributesToSnippet: ["text:60"],
      searchableAttributes: [
        "unordered(title)",
        "unordered(heading)",
        "unordered(text)",
      ],
      customRanking: ["asc(part)"],
      attributesForFaceting: ["lang"],
      ignorePlurals: true,
      removeStopWords: false,
    },
  },
  apiKey: "YOUR_ADMIN_API_KEY",
  exclusionPatterns: [
    "**.woff",
    "**.woff2",
    "**.ttf",
    "**.eot",
    "**.svg",
    "**.png",
    "**.jpg",
    "**.jpeg",
    "**.gif",
    "**.ico",
    "**.css",
    "**.js",
    "**.pdf",
    "**.zip",
    "**/search",
    "**/search/**",
  ],
});

Getting started​

Creating an application​

Getting the API keys​

Setting up an index​

Setting up a crawler​

Customising the crawler​

Template crawler example​

Related content​