How would Microsoft’s Copilot make a Knowledge Enrichment API?

3 min readFeb 16, 2024

So, I asked Microsoft’s Copilot (via Bing):

How would you make a knowledge enrichment API by using Wikipedia dump and diluted sample of the web using common crawl’s dataset? Think of it like a professional in the field with over 25 years of experience.

The answer from Copilot:

That’s a very interesting and challenging question. A knowledge enrichment API is a web service that can provide additional information or insights about a given topic, entity, or concept. For example, given a query like “Barack Obama”, a knowledge enrichment API could return facts such as his date of birth, his political party, his awards, his family members, and so on.

To make a knowledge enrichment API by using Wikipedia dump and diluted sample of the web using common crawl’s dataset, I would follow these steps:

- First, I would download the Wikipedia dump¹ and the common crawl dataset² and store them in a distributed file system, such as Hadoop or Amazon S3.
- Second, I would preprocess the data by extracting the text content, the links, the categories, and the infoboxes from the Wikipedia articles, and by filtering out the irrelevant or low-quality web pages from the common crawl dataset. I would also normalize the text by removing punctuation, stopwords, and stemming the words.
- Third, I would build a knowledge graph from the data by identifying the entities, the attributes, and the relations among them. I would use natural language processing techniques, such as named entity recognition, entity linking, relation extraction, and coreference resolution, to extract the structured information from the unstructured text. I would also use external sources, such as DBpedia³ or Wikidata⁴, to enrich the knowledge graph with more facts and links.
- Fourth, I would index the knowledge graph and the text data by using a search engine, such as Elasticsearch or Solr, to enable fast and scalable retrieval of the information. I would also use techniques such as inverted index, term frequency-inverse document frequency (TF-IDF), and vector space model, to rank the results based on their relevance and importance.
- Fifth, I would design and implement the API by using a web framework, such as Flask or Django, to expose the functionality of the knowledge enrichment service. I would define the endpoints, the parameters, the formats, and the documentation of the API. I would also implement the logic to handle the user queries, to query the search engine, to process the results, and to return the response in JSON or XML format.

This is how I would make a knowledge enrichment API by using Wikipedia dump and diluted sample of the web using common crawl’s dataset. Of course, this is a simplified and high-level overview of the process, and there are many details and challenges that I have not covered. But I hope this gives you some idea of the steps and the techniques involved in this task. 😊

If you want to learn more about web APIs, how they work, and how you can use them in your code, you can check out this tutorial⁵ from MDN Web Docs. It covers the basics of web APIs, such as HTTP requests and responses, JSON, XML, and RESTful principles. It also provides some examples of using web APIs with JavaScript and other languages..

How would Microsoft’s Copilot make a Knowledge Enrichment API?

Written by Tsubasa Kato

No responses yet