# A Guide to Collecting, Processing and Analyzing Online Job Ad Data

<figure><img src="/files/Nvae6xhWnYTLdUBRxLIg" alt="Logos of Bertelsmann Stiftung and Bundesinstitut für Berufsbildung"><figcaption></figcaption></figure>

Online job ads (OJA) can provide detailed labor market insights relevant to the public, private, and civil society sectors. This project highlights which challenges arise in analyzing OJAs, which methods allow for a meaningful interpretation, and which quality criteria to report for transparency purposes. Furthermore, we want to highlight experts in this field and good practices. The content is geared towards the German context, but most of the insights should be transferable to other countries.

### About This Project

The project was initiated by Bertelsmann Stiftung and the Federal Institute for Vocational Education and Training. The project manager and main author is Johannes Müller ([\&effect data solutions](https://www.and-effect.com/)) who has extensive experience in working with online job ad data.&#x20;

The project results are published via printable and interactive reports under an open-source license. The interactive website allows the content to develop further over time and remains open for feedback and suggestions.

**Thank you very much** to everyone who has already contributed or provided feedback to the project. This includes among others the Federal Institute for Vocational Education and Training (BIBB), the Institute for Employment Research (IAB), the HRM & Employment Relations team at the University of Innsbruck, the Stellenmarktmonitor team at the University of Zurich,  and the IW Cologne.

### License

This work is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

![](/files/X1g3ieSGdVYAE2nEzcrG)

### Navigate the Guide

The guide is designed for quick navigation between the sections. There are different ways to read the guide: The recommended way is to follow the path of one online job posting through the whole process until it is included in the analysis: In this project, we call it the OJA [Lifecycle](/navigation/lifecycle).

Alternatively, if you have a direct question or challenge, you can navigate to the related section at [Overview - Challenges](/navigation/overview-challenges). To get an overview of all methods discussed, start with the [Overview - Methods](/navigation/overview-methods).&#x20;

{% embed url="<https://www.youtube.com/watch?v=lZw-1dSDK7g>" %}
A quick overview over the project and the reasoning behind the design decisions by the author Johannes Müller
{% endembed %}

### What Can We Learn from Online Job Ads?

Online job ad (OJA) analysis is the process of analyzing job advertisements posted online. The goal is to understand the skills, qualifications, and requirements that employers most demand. This can be done using automated tools that can analyze large volumes of job ads and identify trends and patterns.

Some of the many reasons why one might want to analyze OJAs include:

* **Identifying the most in-demand skills and qualifications**: By analyzing job ads, it is possible to identify the skills and qualifications that employers most often mention. This can help job seekers understand what they need to focus on to be competitive in the job market. Furthermore, it can help education providers develop evidence-based programs.
* **Understanding the types of positions that are available**: By analyzing the job titles and descriptions in job ads, it is possible to get a sense of the kinds of positions that are available and the industries in which they are located.
* **Identifying any trends or changes in the job market**: By analyzing job ads over time, it is possible to identify any changes or trends in the job market. For example, suppose the number of job ads for a particular skill or qualification increases or decreases significantly. In that case, this may indicate a change in the demand for that skill or qualification.
* **Comparing job ads across different locations or industries**: By comparing job ads from different areas or industries, it is possible to get a sense of the differences in the types of positions and qualifications sought in other parts of the country.
* **Understanding discriminatory language in job ads**: Identifying language-based barriers to the attraction of job seekers generally and of particular groups (e.g., women, older workers, people with migration backgrounds). Terminology of (online) job ads is shaped by bias, excluding qualified job seekers from applications and leading to the perpetuation of inequalities and labor shortages. For example, a matching of job ads with *bias dictionaries* allows for predicting effects on reactions to job ads and application outcomes of job ads.

Overall, online job ad analysis can be a valuable tool for job seekers, employers, and researchers looking to understand the job market and make informed decisions about their career or recruitment efforts.

### What Are the General Limitations of Online Job Ads?

While an important tool to get insights into the labor market, OJAs have several limitations:

* **Bias**: OJAs do not represent the labor market at large as they only capture positions that are advertised online. Many open positions are not published online and on the other hand, job seekers might not only look online for postings. The factors that determine which jobs get published might vary depending on the industry, organization size, etc.
* **Incomplete Information**: OJAs can only provide a summary of tasks and responsibilities for a position and therefore often provide an incomplete list of the qualifications actually required.
* **Duplication**: As positions are often advertised on multiple platforms or websites at the same time, it makes it difficult to get an accurate estimate of how many unique job postings there are at a given time.
* **Expiration of job postings**: Job postings sometimes remain active on job portals even though the position is already filled. Furthermore, there are several challenges involved in collecting data continuously to identify when exactly a job posting was deactivated. This further complicates the issue.
* **Misleading language**: Job postings are written in order to attract potential candidates and meet legal requirements. This can result in misleading language and makes it more difficult to accurately capture the qualifications required.

Some of these limitations can be mitigated and are described in [Overview - Challenges](/navigation/overview-challenges), but some limitations are inherent to any analysis of OJA data.

> Note that employers do not disclose all aspects of a given job in the job ad. For instance, [Stops et al. (2021, p. 21)](/in-practice/literature-and-projects) assume that employers include information like skill requirements explicitly in situations when these employers consider them to be essential for carrying out the activities in the job they offer.&#x20;
> &#x20;They also mention further criteria for including information in job ads. They conclude that, though job ads rarely represent complete job profiles, this information is highly relevant for the recruitment of workers for the respective job.
>
> *Michael Stops (IAB)*


# Lifecycle

An introduction to the Online Job Ads Analysis Cycle.

## The OJA Lifecycle

The process of analyzing OJAs is not sequential. However, at some point or another, online job ads undergo implicitly or explicitly a process from data collection to data analysis:

1. **Unstructured data**: At the beginning of the analysis cycle, an online job ad is published in a job portal, on a company website, or in a digital news publication. The data at this point consists of different web elements and is more or less unstructured.&#x20;
2. **Structured data**: After collecting the data through, e.g., web scraping or calling an API, the next step is to structure the data by extracting relevant information from the job ad and organizing it. This may involve creating a spreadsheet or database with columns for different information such as job title, salary, full text, text segments, etc.&#x20;
3. **Enriched data**: Once the data is structured, it can be enriched by adding additional information or extracting information from the text. We are often interested in analyzing these enriched concepts: Job titles, competencies, soft skills, education level, etc. Often these concepts need to be linked to a formalized taxonomy.&#x20;
4. **Filtered dataset**: After the data has been structured and enriched, it can be used to create a sampling dataset. This step involves deduplication, representativity analysis, and other considerations.
5. **Analysis**: The final step is to analyze the sampling dataset to identify trends and patterns. Depending on the goals or the data structure, we can use tools such as inferential statistics, hypothesis testing or text mining.

These steps are almost never followed linearly. For example, the data preparation step may need to be automated or repeated if new job ads are added, or if you already have a structured dataset for your analysis. However, at some point or another, your final analysis will have included all of these steps.

Making these steps more transparent and highlighting decisions and trade-offs along the way is the goal of this publication.

## Step by Step

The OJA analysis process presents various challenges at each step. However, there are multiple methods and approaches available to overcome these challenges. The visualization below illustrates the stages that an OJA dataset goes through, from being collected in an unstructured format to being transformed into a useful, cleaned, and enriched dataset ready for analysis, while highlighting the corresponding steps of the OJA analysis process.

<figure><img src="/files/xVwPBVRSKr5OFhNr1w7q" alt=""><figcaption><p>Visualization of the OJA cycle: From data collection and unstructured data to an organised, cleaned and enriched dataset ready for analysis.</p></figcaption></figure>


# Overview - Challenges

This page gives an overview of common challenges you could encounter when analyzing OJAs. Next to each challenge is the relevant section in the guide.

{% hint style="info" %}
If you have identified any challenges not listed here, please let us know by making a pull request in our GitHub Repository or contact us directly.
{% endhint %}

## Data Collection

* Where to find OJA data? [Data Collection](/steps/data-collection#data-sources-and-oja-landscaping)
* How to collect OJA data? [Data Collection](/steps/data-collection#web-scraping)
* How to store OJA data? [Data Collection](/steps/data-collection#job-posting-data-schema)

## Data Enrichment

* How to segment a job ad? [Data Enrichment](/steps/data-enrichment#text-segmentation)
* How to identify duplicates? [Data Enrichment](/steps/data-enrichment#identifying-duplicates)
* How to extract occupations? [Data Enrichment](/steps/data-enrichment#normalising-job-titles)
* How to extract skills and competences? [Data Enrichment](/steps/data-enrichment#extracting-skills)

## Evaluation and Quality Control

* How to evaluate extraction and classification algorithms? [Evaluation and Quality Control](/steps/evaluation-and-quality-control#evaluation-metrics)
* How to create a gold standard for evaluation? [Evaluation and Quality Control](/steps/evaluation-and-quality-control#gold-standard-annotation-and-quality)

## Taxonomies and Ontologies

* Which taxonomies are there (ISCO, ESCO, KLDB, etc.)? [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#taxonomies-for-online-job-ad-analysis)
* How to develop a taxonomy? [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#developing-a-taxonomy)
* How to evaluate a taxonomy? [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#data-standards-for-taxonomies-and-ontologies)

## Dataset Curation and Representativity Analysis

* How to deal with duplicates? [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#deduplication)
* How to validate an OJA dataset/sample? [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#representativity-analysis)
* What to report?  [Evaluation and Quality Control](/steps/evaluation-and-quality-control#reporting-results)


# Overview - Methods

This page gives an overview of methodological approaches in the field of OJA analysis. Each method is linked to its respective section in this guide.

## Data Collection

* Landscaping [Data Collection](/steps/data-collection#data-sources-and-oja-landscaping)
* Web Scraping [Data Collection](/steps/data-collection#web-scraping)
* API Data Collection [Data Collection](/steps/data-collection#api-data-collection-data-providers)
* Data Formats [Data Collection](/steps/data-collection#job-posting-data-schema)

## Data Enrichment and Methods

* Text Segmentation [Data Enrichment](/steps/data-enrichment#text-segmentation)
* Duplicate Identification [Data Enrichment](/steps/data-enrichment#identifying-duplicates)
* Natural Language Processing and Data Pre-Processing [Extraction Methods](/steps/extraction-methods#pre-processing-and-embeddings)
* Rule-based Matching [Extraction Methods](/steps/extraction-methods#rule-based-matching)
* Supervised Document Classification [Extraction Methods](/steps/extraction-methods#supervised-classification)
* Named-Entity Recognition [Extraction Methods](/steps/extraction-methods#statistical-named-entity-recognition-token-classification)
* Disambiguation/ Entity Linking [Extraction Methods](/steps/extraction-methods#semantic-similarity)

## Evaluation and Quality Control

* Gold Standard Annotation [Evaluation and Quality Control](/steps/evaluation-and-quality-control#gold-standard-annotation-and-quality)
* Machine Learning Evaluation [Evaluation and Quality Control](/steps/evaluation-and-quality-control#evaluation)

## Taxonomies and Ontologies

* Taxonomy Development [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#developing-a-taxonomy)
* Taxonomy Evaluation [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#data-standards-for-taxonomies-and-ontologies)

## Dataset Curation and Representativity Analysis

* Sampling [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#filtering-data)
* Deduplication [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#deduplication)
* Representativity Analysis [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#representativity-analysis)


# Data Collection

Collecting and representing OJA data.

There are several data sources for online job ad data. The most common is web scraping of employers' websites, job boards, or aggregation portals. Some job or aggregation portals offer APIs to access the data directly.&#x20;

The process of how data is stored, collected and processed directly influences the quality of the data, its representativity, and its usefulness. Therefore, a solid understanding of the processes is crucial for interpreting any results down the line.

### Data Sources and OJA Landscaping

The study of where online job ads are published is worthy of investigation, as the online job matching market is growing and changing quickly. [Cedefop](https://www.cedefop.europa.eu/en) and Eurostat carry out [country-specific OJA market analyses](https://www.cedefop.europa.eu/en/country-reports), which aim to identify essential job portals that would ensure representative coverage of the system and provide context for interpreting country data.

{% embed url="<https://youtu.be/AR01w5r8DdY>" %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/employees-wanted-analysis-of-online-job-advertisements-a-series-2-6/>" %}
Link to the blogpost on OJA landscaping by Jiri Branka
{% endembed %}

### Web Scraping

After determining where OJAs can be found, the question is how can the data be collected? The most common technique for this purpose is *web scraping*. Web scraping refers to the process of extracting data from websites.&#x20;

In the context of online job analysis, this means that individual job postings can be targeted and automatically downloaded. The process of scraping OJAs is supported on many websites by the implementation of specific data schemas - which for example guide aggregator platforms to pick up on the job ads. While usually referring to downloading targeted job ads, some broader methods aim at identifying relevant job ads themselves.&#x20;

This process is often called *spidering* or *crawling*. Here is how TextKernel, a relevant OJA data provider, describes spidering in the context of their jobfeed:

> Jobfeed obtains new jobs from the Internet daily through spidering. In order to achieve broad and deep coverage, Jobfeed uses two spider methods: wild spidering and targeted spidering.\
> The wild spider is a system that works automatically and dynamically. It continuously indexes hundreds of thousands of relevant (company) websites and discovers new job postings.
>
> Targeted spider scripts are created to retrieve jobs from specific – usually large – websites, like job boards, and websites of large employers. Despite their size and complexity, the script ensures that all jobs are found. The targeted spider scripts run multiple times per day.
>
> [*Textkernel: How Jobfeed by TextKernel works*](https://www.textkernel.com/technology/how-jobfeed-by-textkernel-works/)

Automatically crawling websites for relevant job postings might be more efficient but also comes with trade-offs in terms of data quality. Here is Cedefop describing the trade-off:

> Crawling uses a programmed robot to browse web portals systematically and download their pages. Crawling is much more general compared to scraping and is easier to develop. However, crawlers collect much more website noise (irrelevant content) and more effort is needed to clean the data before further processing.
>
> [*Cedefop, Online job vacancies and skills analysis: a Cedefop pan-European approach, Publications Office, 2019*](https://data.europa.eu/doi/10.2801/097022)

It is important to ensure that the data is collected and used ethically and legally. As web scraping consists sometimes somewhat of a "grey area", it is crucial to ensure that the data collected is used in a way that respects the privacy of individuals and complies with data protection laws. In the German context, the Science Center Berlin commissioned a report from the University of Würzburg on web scraping in independent scientific research in 2018. It comes to the conclusion, that scientists can certainly use the web scraping process from a legal perspective to support their non-commercial research. However, they must consider certain requirements to comply with the law.

{% file src="/files/9shx2EBPtGmvqWK6mtcB" %}
Web Scraping in der unabhängigen wissenschaftlichen Forschung - Gutachten im Auftrag des Wissenschaftszentrums Berlin für Sozialforschung gGmbH (WZB) – Rat für Sozial- und Wirtschaftsdaten (RatSWD) - Paul Vogel, Prof . Dr . Dr . Eric Hilgendorf, 2018
{% endfile %}

Additionally, to data privacy and data protection considerations, some websites have terms of use that prohibit or limit web scraping - which can be found either in their terms of service or in a [robots.txt file](https://www.oreilly.com/library/view/python-web-scraping/9781786462589/9ee32cdd-619b-4f4b-a812-f796276548e7.xhtml).&#x20;

{% hint style="info" %}
An often overlooked challenge in web scraping is maintainability. While it might be suitable for one-of analysis of a limited sample, maintaining scrapers, crawlers, and spiders over a longer period of time needs considerable resources. Websites change their schemas and URLs. Furthermore, data quality must be monitored closely.\
While web crawling might seem cheaper compared to buying data from data providers, however, the costs and resources needed for web crawling at scale cannot be underestimated.
{% endhint %}

{% embed url="<https://github.com/UKDataServiceOpen/web-scraping>" %}

### API Data Collection / Data Providers

The main challenges with web scraping can sometimes be mitigated by using APIs supported and advertised by OJA data providers. Generally, APIs (Application Programming Interfaces) allow different software systems to communicate with each other and exchange data. APIs allow developers to access and retrieve data from their systems in a controlled and programmatic way. The data accessible over APIs is generally high in quality as they are generated through a controlled web scraping process (with data quality checks) or come directly from structured databases of job portals.

Some API providers also enrich their data to increase its value by attaching standardized job titles, extracting skills or providing data on the employer.

Examples for a job data APIs are the [TextKernel JobFeed API](https://developer.textkernel.com/JobfeedAPI/master/api/) or the[ Lightcast Burning Glass API](https://api.lightcast.io/guides/bgt-api-suite).

{% hint style="info" %}
Some data providers may only provide meta data and not the full text data. For your project or product you might want to carefully consider what your requirements are for data quality, your need to develop and employ your own enrichment algorithms and the amount of data you need for analysis.
{% endhint %}

Using APIs usually requires authentication through an API key or access credentials which require a subscription to the service. Depending on the service offering, APIs can limit the number and frequency of requests that can be made and have other terms of use that must be followed.&#x20;

{% hint style="info" %}
While web crawling might seem cheaper compared to the high costs of buying data from data providers, however, the costs and resources needed for web crawling at scale cannot be underestimated.&#x20;
{% endhint %}

### Job Posting Data Schema

When collecting OJA data and storing it, it is important to consider how job postings are represented as data and how they can be stored in a structured way.

Job postings often have a set of properties and meta data such as title, description (full text), occupation, employment type, location, etc. Every portal or website has its own way of structuring this information. Therefore, there is not one unified schema for representing OJA data.&#x20;

The closest to a universal specification of a job posting data schema is the `JobPosting` type provided by [schema.org](https://schema.org/JobPosting). schema.org is a collaborative community activity that aims to create, maintain, and promote unified data schemas for different kinds of data.&#x20;

Here is an example of how a job posting might be marked up using the `JobPosting` type:

```markup
<div itemscope itemtype="http://schema.org/JobPosting">
  <h2 itemprop="title">Software Engineer</h2>
  <span itemprop="description">
    We are seeking a highly skilled software engineer to join our team. 
    The successful candidate will have experience building and 
    maintaining software applications, as well as a strong understanding
    of computer science principles.
  </span>
  <div itemprop="hiringOrganization" itemscope itemtype="http://schema.org/Organization">
    <span itemprop="name">ACME Corp</span>
  </div>
  <div itemprop="employmentType" itemscope itemtype="http://schema.org/EmploymentType">
    <span itemprop="name">Full-time</span>
  </div>
  <div itemprop="jobLocation" itemscope itemtype="http://schema.org/Place">
    <span itemprop="address" itemscope itemtype="http://schema.org/PostalAddress">
      <span itemprop="addressLocality">San Francisco</span>,
      <span itemprop="addressRegion">CA</span>
    </span>
  </div>
  <div itemprop="baseSalary" itemscope itemtype="http://schema.org/MonetaryAmount">
    <span itemprop="currency">USD</span>
    <span itemprop="value" itemscope itemtype="http://schema.org/PriceSpecification">
      <span itemprop="minValue">100000</span>
      <span itemprop="maxValue">120000</span>
      <span itemprop="unitText">YEAR</span>
    </span>
  </div>
  <div itemprop="validThrough" content="2022-01-01">December 1, 2023</div>
</div>
```

By using the `JobPosting` type and including these properties, OJA providers help search engines and other crawlers or scrapers structure OJAs. [Here](https://developers.google.com/search/docs/appearance/structured-data/job-posting) is an example of the schema explained by Google for better integration with their job ad indexing service.

When designing your own class for representing job posting data it is usually a good idea to either use the schema.org definition directly or something close to it to ensure interoperability. Here is an [example of a posting in the JSON format](https://workforce-data-initiative.github.io/skills-ml/common_schema/) developed by the University of Chicago, which also draws from the schema.org specification:

```json
{
    "incentiveCompensation": "",
    "experienceRequirements": "Here are some experience and requirements",
    "baseSalary": {"maxValue": 0.0, "@type": "MonetaryAmount", "minValue": 0.0},
    "description": "We are looking for a person to fill this job",
    "title": "Bilingual (Italian) Customer Service Rep (Work from Home)",
    "employmentType": "Full-Time",
    "industry": "Call Center / SSO / BPO, Consulting, Sales - Marketing",
    "occupationalCategory": "",
    "qualifications": "Here are some qualifications",
    "educationRequirements": "Not Specified",
    "skills": "Customer Service, Consultant, Entry Level",
    "validThrough": "2014-02-05T00:00:00",
    "jobLocation": {"@type": "Place", "address": {"addressLocality": "Salisbury", "addressRegion": "PA", "@type": "PostalAddress"}},
    "@context": "http://schema.org",
    "alternateName": "Customer Service Representative",
    "datePosted": "2013-03-07",
    "@type": "JobPosting"
}
```

### Data Storage

As discussed above, storing OJA data comes with its own challenges. In general it is advisable to store large amounts of data in a database. There are three major types of databases that all have their advantages and disadvantages when dealing with OJA data:

**Relational Databases:** Relational databases (such as SQL databases) are well-established and widely used. Since the data is structured in clearly defined schemas, it is easy to work with very complex queries. On the other hand, the use cases should be rather well-defined before setting up the database as any changes will require changes to its schemas.

**Document Databases**: Often also called No-SQL-Databases (such as MongoDB), these were designed for semi- or unstructured data. Document databases are by nature flexible and scalable, allow for storing OJAs as they are collected. Compared to SQL databases they are not efficient when it comes to querying and data quality must be observed more closely.

**Graph Databases**: OJAs can be represented as graphs since postings usually share many properties (such as organizations, skills, occupations, etc.). In relational databases, these relations would be linked between the tables. In graph databases, OJAs are represented as a graph network. This makes very complex queries possible and provides a high level of data consistency and data integrity. The power of graph databases in the context of OJA is exemplified by the ontologies provided by the European Union (such as ESCO and ISCO), which are inherently structured as linked open data.

Which database type to choose depends to a great extent on the specific requirements and use cases. While the "natural" environment for semi-structured documents like OJAs is document databases, the other two types usually provide a better interface for large-scale, structured analysis.

{% hint style="info" %}
Storing OJA data in a tabular format such as .csv or .xlsx is challenging as most properties are nested by nature. For example, one job posting might have multiple related skills or occupations. A workaround might be to store information in separate tables that can be joined using a common key. In practice, this could look as follows:

* a **meta table** containing rows with job postings. Each posting is identified using a unique identifier. This table contains information that has clear 1:1 relationships with the posting (when it was published, on which website it was published, etc.)
* a **skills table** containing rows with skills. Each skill is mapped to a posting id from the meta table.

This approach however does not scale well and is memory and processing intensive for large datasets.
{% endhint %}


# Data Enrichment

Enriching OJA data by extracting relevant concepts from the collected data.

After collecting OJA data from primary or secondary sources, it is often needed to enrich the data for analysis. In very few cases, the indicators and properties we are interested in analyzing are already available in a structured way.

This page is closely related to the [Extraction Methods](/steps/extraction-methods)section but focuses more on specific indicators and properties of OJAs.&#x20;

{% hint style="info" %}
The term "enrichment" was chosen for this section as the goal of each of the following steps is to add more structure or new information to an OJA. Duplicate identification is a good example of this framing: The identification of duplicates usually involves adding a separate indicator to a posting and linking it to other postings, rather than directly removing them from a dataset. Dealing with duplicates (e.g. removing them for analysis) then constitutes another process that is related to [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis).
{% endhint %}

### Text Segmentation

Text segmentation (or "zoning") refers to identifying and classifying paragraphs or text segments in OJAs. Among others, the following segments can be found in many OJAs:&#x20;

* description of the company/employer
* description of the job (e.g., tasks)
* required candidate profile
* description of benefits
* contact fields

Segmenting the full text of an OJA can have multiple advantages for downstream extraction or analysis tasks. One example is that segmenting text can drastically increase the performance of rule-based extraction methods.

> Job ads contain information on topics such as the company, the job, or required qualifications. For an accurate extraction of skills and tasks, we need to identify the corresponding text zones, as many key terms are ambiguous, for instance ‘dynamic’ might refer to a personality trait or to a dynamic CRM system.
>
> [*Gnehm, Ann-Sophie, und Simon Clematide. 2020. „Text Zoning and Classification for Job Advertisements in German, French and English“. S. 83–93 in Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science. Association for Computational Linguistics.*](https://aclanthology.org/2020.nlpcss-1.10.pdf)

Another example is to semantically differentiate meaning in job ads by segmenting.

> "Job ads serve both a selection and a marketing function. Analyzing job ads thus requires the segmentation of job ad texts along these functions, with employer description and offer/benefit referring to the marketing function and job description and candidate profile referring to the selection function. Research has shown that a word used in different text segments of job ads leads to different results in applicant attraction. For instance, whereas "flexible" as a candidate's trait appears more attractive for men than for women, "flexible" as an attribute of the employing organization or the work context (as, e.g., in "flexible work hours") is more attractive for women than for men."
>
> *Julia Brandl and Petra Eggenhofer-Rehart*\
> *More details in Eggenhofer-Rehart, P., Brandl, J., & Kohlberger, M. (2022). Flexibility and flexibility are not the same: A genre-sensitive method to measuring gendered wording in job advertisements. Vortrag am Herbstworkshop der Wissenschaftlichen Kommission Personalwesen im Verband der Hochschullehrer für Betriebswirtschaft e.V., Berlin, Germany, 29-30 September 2022)*

As online job ads are usually very structured text documents, good results can be achieved: The [Gnehm and Clematide (2020)](https://aclanthology.org/2020.nlpcss-1.10.pdf) study achieved around 90 percent accuracy, [Murauer et al. (2018)](http://www.cicling.org/2018/intranet/pre-print/papers/paper_12.pdf) achieved  78 percent accuracy, and [Hermes and Schandock (2016)](https://www.bibb.de/dienst/veroeffentlichungen/de/publication/show/8146) got results of up to 97 percent accuracy (F-Score of .95). A viability study by [Stops et al. (2021)](https://doku.iab.de/forschungsbericht/2021/fb0721.pdf) for extracting skill requirements achieved 98 percent accuracy (F-Score of .98) on this task. It is important to note, however, that the performance of text segmentation algorithms can vary significantly based on the overall data quality and data source. The study results above are therefore not directly comparable and need to be assessed in the specific task they were trained on. Furthermore, the generalization of these models to other datasets is challenging.

{% embed url="<https://youtu.be/TQ_nw2bdMq8>" %}
Presentation by Dr. Felix Busch on Text-Zoning at the OJV Forum 2021
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-aufbereitung-eines-datenschatzes/>" %}

### Identifying Duplicates

Duplicates are job postings that refer to the same vacancy. Identifying duplicates of job postings is crucially important for large-scale analysis of online job ads. If there are duplicates in the data set, it can result in the analysis being skewed or misleading. For example, if the same job is included multiple times in the data set, it appears as though there is a higher demand for that particular type of job than there actually is. [TextKernel estimates](https://www.textkernel.com/newsroom/online-job-postings-have-many-duplicates-but-how-can-you-detect-them-if-they-are-not-exact-copies-of-each-other/) that on average a job ad is reposted two to five times, which leads to 50 to 80 percent of job postings being duplicates.

In order to remove duplicates from a dataset we first have to identify them. This is a challenging task as jobs might be advertised on multiple job sites which are not identical in structure or content. Furthermore jobs might be altered slightly when published in different days. A different challenge is that similar job posting might occur weeks or months apart and represent different vacancies while being semantically very similiar.&#x20;

> "Another tough test for models for duplicate detection would be job postings from a company with different job locations. Linguistically, the job postings could be identical except for the job location. This example makes it clear that a simple examination of text similarity is not sufficient, but features such as job location, employer, and deadline must be extracted from the text, as described in the next paragraph."
>
> *Stefan Winnige (BIBB)*

To mitigate these challenges, multiple methods are usually used to identify duplicates: The process often involves an evaluation of the [Extraction Methods](/steps/extraction-methods#semantic-similarity) and the similarity on indicators from the meta data (publishing date, location, organization, job title).

Some results indicate that duplicate identification can achieve very good results with accuracy and F-scores of .95 and above ([Zhao et al. 2021](https://www.researchgate.net/profile/Claire-Mason-9/publication/359256401_Zhao_Chen_Mason_duplicate-detection-ACM/links/623186550837bf2b9edacb76/Zhao-Chen-Mason-duplicate-detection-ACM.pdf)). However, results again depend heavily on which dataset is used and its data quality ( see [Data Collection](/steps/data-collection) ). Furthermore, larger datasets make duplicate identification computationally expensive and increasingly hard to do. [TextKernel summarizes their modeling efforts](https://www.textkernel.com/newsroom/online-job-postings-have-many-duplicates-but-how-can-you-detect-them-if-they-are-not-exact-copies-of-each-other/) in the following way: "How accurate is the system \[...]? Does it find all duplicates? Does it cluster together postings that are not duplicates? A short answer would be that the system is 'pretty good, 90%-ish'. A longer answer would require a discussion of many possible ways to evaluate such a system".&#x20;

Refer to the section on [Dataset Curation and Representativity Analysis](/steps/dataset-curation-and-representativity-analysis#deduplication) for strategies to remove identified duplicates from a dataset.

{% embed url="<https://youtu.be/zQTyxyuPL30>" %}
Presentation by Anne Gerlach on duplicate identification at the OJV Forum 2021.
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-erkennung-von-dubletten/>" %}
Blog Post about Duplicate Identification.
{% endembed %}

### Normalizing Job Titles

Normalizing a job title means converting it to a standardized term or category that is used in the taxonomy. For example, the job title "We look for a data science expert" should be linked to the standardized profession "[2511.4 - data scientist](https://esco.ec.europa.eu/en/classification/occupation?uri=http%3A%2F%2Fdata.europa.eu%2Fesco%2Foccupation%2F258e46f9-0075-4a2e-adae-1ff0477e0f30)" in the ESCO classification.&#x20;

This process is important because it helps to ensure that job titles are consistently represented and understood, regardless of the specific wording or phrasing used. Without normalization, it can be difficult to analyse online job ads in a structured way.

There are a myriad of methods that can be used for job title normalization like [Extraction Methods](/steps/extraction-methods#rules-based-matching), [Extraction Methods](/steps/extraction-methods#classification), or [Extraction Methods](/steps/extraction-methods#semantic-similarity) matching. No matter the method, normalizing job titles is a very challenging task because of semantic and construct ambiguity. Some challenges that might arise are:

* a job title can refer to multiple occupations; e.g. "Helfer/in im Bau" ("Assistant in construction" without specifying which area of construction)
* a job title mentions multiple occupations; e.g. "Hochbauhelfer/in oder Tiefbauhelfer/in", "Service- und Küchenkraft" ("Assistant for building construction or underground construction"; "Service and kitchen staff")
* a job title can be unspecific; e.g. "Projektleiter/in Elektro" (Project manager for electrical works)
* the taxonomy itself is too detailed for ambiguous job titles; e.g. "Software Developer" and "Frontend Developer" might be two different occupations in a taxonomy, where a job title might not make a distinction.

Job title normalisation models are exceedingly hard to evaluate using traditional metrics. This has to do with the reasons mentioned above, but also with some other challenges: Gold standard datasets that are created synthetically might have issues when transferring them to out-of-sample distributions. Gold standards created from online job postings need heavy preprocessing to ensure a certain class balance (some occupations are just very rare). Even then, taxonomies need to be updated regularly and so do extraction models to ensure that job titles can actually be normalized.

{% hint style="info" %}
Developing an extraction model for a certain sector or a limited number of occupations might be easier to achieve - for three reasons: 1) It is easier to create balanced gold standard datasets, 2) variance is limited (and data distributions don't incorporate the whole job market), 3) it is easier to incorporate domain knowledge for a limited number of occupations.
{% endhint %}

Which [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies) or classification scheme is used depends mostly on the context of analysis and has vast implications for the design and the downstream performance of the normalisation approach. Often taxonomies are expanded to more complete ontologies that incorporate domain knowledge through synonyms, lexicalizations and relations.&#x20;

> Job postings and taxonomies have different purposes. This conflict cannot be entirely resolved.However, you can be aware of them and use this awareness to guide your interpretation of the results.
>
> [*Claudia Plaimauer - OJV Forum 2021*](https://blog.aus-und-weiterbildung.eu/oja-zuordnung-von-berufen/)

{% embed url="<https://youtu.be/PE1yX8V8mak>" %}
Presentation by Claudia Plaimauer on building an occupation ontology at the OJV forum 2021.&#x20;
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-zuordnung-von-berufen/>" %}
Blog Post by Claudia Plaimauer on building an occupation ontology.
{% endembed %}

### Extracting Skills

After the job title, skills and competences in OJAs are the most used properties for analysis. It is important to note that there are many different definitions of what a skill actually constitutes, and it is a highly context-specific construct.

On a technical level, the goal of skill extraction is to identify and classify word sequences in text that relate to skills as defined in the analysis.&#x20;

As with job titles there are two general approaches one can employ: The first is an ontological approach where a skill taxonomy is carefully curated and enriched with search words, synonyms, lexicalizations and relations. This information is used to build a [Extraction Methods](/steps/extraction-methods#rules-based-matching)model.

The second approach would be a statistical extraction of skills. Here we can differentiate between two steps.

* **Skill Localization**: In this step the word sequence boundaries are found in the text - usually using a statistical model like [Extraction Methods](/steps/extraction-methods#statistical-named-entity-recognition-token-classification). This generates a list of skills that are not yet mapped to a taxonomy. This step might be split up into two smaller steps where first skill candidates are generated and subsequently classifed as containing a skill or not ( [Extraction Methods](/steps/extraction-methods#classification)).
* **Skill Disambiguation**: Taken a list of skills the concepts then have to be matched to a taxonomy. This can involve the classification to concepts of a classification scheme or taxonomy or can be down bottom-up using clustering and unsupervised learning. This step is similar to the normalisation of job titles and can involve [Extraction Methods](/steps/extraction-methods#semantic-similarity), [Extraction Methods](/steps/extraction-methods#rules-based-matching) or [Extraction Methods](/steps/extraction-methods#classification).

{% embed url="<https://youtu.be/mkIWcmApLuw>" %}
Presentation of the Kompetenz-Kompass (Stops 2021) by Dr. Michael Stops at the OJV Forum 2021.
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/gewusst-wie-analyse-voja-genese-und-anreicherung-von-kompetenzwoerterbuechern/>" %}
Blog post about the Kompetenz-Kompass.
{% endembed %}

### Other Indicators

There are many more indicators that could be of interest for OJA analysis: Data on wages, remote work options, sustainability, company presentation, and many more.

To give just two examples: To analyze gender bias in OJAs, predominantly male-stereotyped words, such as "high-performing" or "career-oriented," can be extracted and enrich the analysis. Another use case is to scrutinize job titles in more detail to derive the rank or seniority of the job within a company. The "Seniority Score" could then be used to suggest suitable job offers to applicants based on their current job title.

These two use cases are described in the following presentation:

{% embed url="<https://www.youtube.com/watch?v=K07NU45uOZQ>" %}
Presentation on indicator enrichment by Dr. Jannis Buchsteiner (Stepstone) at the OJV Forum in 2021.
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-aufdecken-von-versteckten-informationen/>" %}
Blog Post about additional indicators.
{% endembed %}

Another example for additional indicators is the extraction of sustainability keywords in OJAs. Analyses like this can help to get a better understanding of the signalling in OJAs and how companies present themselves.

The following video describes the approach by Johanna Binnewitt and Timo Schnepf.

{% embed url="<https://www.youtube.com/watch?v=YtUpSKPbtIU>" %}
Presentation by Johanna Binnewitt and Timo Schnepf at the OJV Forum 2021.
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-werben-mit-nachhaltigkeit-in-azubistellen/>" %}
Blog Post about extracting sustainability keywords from OJAs.
{% endembed %}


# Extraction Methods

Overview of extraction methods commonly used in OJA enrichment processes.

## Pre-Processing and Embeddings

### Embeddings

Text embedding is a technique used to represent text in a numerical format that can be input into a machine learning model or used for analysis. Most natural language methods rely on embedding methods as they transform natural language text into something that can be used in statistical methods. While often being a pre-processing step, examples, where word embeddings are used directly, are semantic similarity tasks or semantic modeling tasks, e.g., for finding similar documents or synonyms in context.

Some standard text embedding techniques are:

1. **Count vectorization**: This technique represents a text as a numerical vector based on the occurrences or frequency of words within the text.
2. **tf-idf** (term frequency-inverse document frequency): This method represents a text as a numerical vector based on the frequency of words within the text and the importance of those words in the overall corpus of text.
3. **Word2Vec**: This method represents words as numerical vectors based on the context in which they appear. Similar approaches are Tok2Vec, Doc2Vec, or GloVe.
4. **Transformer models**: These are a newer type of text-embedding technique based on deep learning models and can capture more complex and contextualized relationships between words. Examples of models that are built on the transformer architecture are BERT and GPT. While you can train transformer models from scratch, it is often easier to take a foundational model and use fine-tuning to adapt it to your data and use case. The only public domain-adapted transformer-based language model for German-speaking job advertisements was published by [Gnehm et al. (2022)](https://huggingface.co/agne/jobBERT-de).

{% embed url="<https://huggingface.co/agne/jobBERT-de>" %}
Published jobBERT model by Gnehm et al. 2022
{% endembed %}

{% hint style="info" %}
While transformer models are the state-of-the-art method for embedding text, traditional methods might work better for some tasks. This can be the case if the modeling goal is straightforward - e.g., if the words themselves are relevant entities rather than the context. If unsure about the complexity of a task, it is usually a good idea to use them as a baseline for comparison.&#x20;

A critical downside of transformer embeddings is that they are computationally expensive and usually require a machine with a GPU to run efficiently.
{% endhint %}

### Tokenization, PoS, Co-Reference Resolution

Apart from text embeddings, you can usually employ other steps before putting the data into a rule-based or statistical model. Each of the techniques represents its own class of models and might be necessary as preprocessing steps (e.g. to compute embeddings).&#x20;

**Tokenization** divides the text into smaller units called tokens. These tokens can be words, phrases, sentences, or other word sequences. Tokenization is often the first step in natural language processing tasks, as it helps to break up the text into manageable pieces that can be more easily analyzed. Splitting texts into sentences is often helpful for the analysis of OJA as some sections of text are highly structured, e.g., bullet points or lists.

> "Especially, the bullet-pointed lists can make it challenging for standard tokenizers to split job ad text into sentences. Another characteristic of job ads are lists that begin with a half-sentence, for example: "You are: - communicative, - team-oriented...". In reality, the job ad contains two sentences, "You are communicative" and "You are team-oriented", which standard tokenizers do not account for."
>
> Stefan Winnige (BIBB)

**Part-of-speech tagging** identifies the part of speech (e.g., noun, verb, adjective) of each word in a text. Part-of-speech tagging can be useful for various natural language processing tasks, such as syntax parsing and text classification. It is also very useful for rule-based matching approaches when identifying, e.g., skills.

**Co-reference resolution** is a method for identifying and disambiguating mentions of the same entity within a text. For example, if a text mentions "the job seeker should..." and "their skills..." in reference to the same person, co-reference resolution would identify these as references to the same entity. Co-reference resolution can be useful for tasks where concepts must be attributed to different entities. For example: Does the adjective "innovative" refer to the employer, or is it an attribute of the job candidate they are looking for?

## Machine Learning Tasks&#x20;

For the modeling and extraction of the concepts in the chapter [Data Enrichment](/steps/data-enrichment), many different approaches can be used. Generally, there is not one gold-standard approach to every problem. Many concepts need a mix of approaches and models.

The following model architectures are not mutually exclusive - Semantic Similarity can be used for classification, or rule-based matching can be combined with a statistical NER model. However, they give a short overview of different generalized modeling approaches for problems often encountered when working with OJA data.

### Document Classification

This model architecture assigns a text document to one or more predefined categories or labels. The model is usually trained on a labeled gold-standard dataset. Classification models can be used for many tasks in OJA analysis, including job title and skill classification. They can also be combined with other approaches for disambiguation or entity linking.

{% hint style="info" %}
One challenge you often encounter in supervised classification is the number of labels or concepts. Multi-class classification models are often designed for a few classes (<10) but not thousands of classes (as with an occupation taxonomy, for example). &#x20;
{% endhint %}

### Token Classification

Named entity recognition (NER) is a popular NLP task that identifies and classifies named entities - traditionally people, organizations, and locations - in a text. NER models are a specific implementation of a broader class of models: token classification models. The main difference to the text classification above is that the classification happens not at the document level but at the token level. Tokens can be words, phrases, arbitrary word sequences, or entire sentences. These models are usually used to identify company names in OJAs, generate "skill candidates" or find other concept mentions in a text.&#x20;

{% hint style="info" %}
Large taxonomies are a challenge for token classification models as well. To mitigate this, you can use a token-level model to identify skill mentions in a text and then use an entity linking or text similarity approach to find the correct concept.
{% endhint %}

### Unsupervised Classification

Unsupervised classification refers to various machine learning techniques to automatically categorize text into categories or labels without explicit training data. This is in contrast to supervised classification, which involves training a model on a labeled dataset.

There are many different architecture and modeling goals in unsupervised learning, like clustering, dimensionality reduction, or topic modeling. Unsupervised learning can also be a technique for preprocessing data for supervised classification.

Unsupervised classification can be useful for online job ad analysis because it allows for automatically grouping text into categories based on its content without manually labeling the data or a formal taxonomy. In fact, it can be used as a vantage point for developing a taxonomy.

### Rule-Based Matching

This type of model architecture uses a set of predefined rules or patterns to identify and extract specific pieces of information from a text. Rule-based matching models can be effective for information extraction and entity-linking tasks. It is an often-used method in online job ad analysis for a couple of reasons: 1) Compared to statistical models, the amount of labeled data required is much lower (usually only for evaluation purposes); 2) when a model is based on a formalized taxonomy, there are often search words and additional information already available; and 3) Using rules and search words gives you more control over the vocabulary than statistical approaches.

{% hint style="info" %}
For a more in-depth discussion on how to build semantic models, look at the section [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#data-standards-for-taxonomies-and-ontologies).
{% endhint %}

On the other hand, rule-based matching has some significant disadvantages: 1) Curating rules is time-intensive, 2) rule-based models don't generalize beyond the rules that were set, 3) rule-based models should be evaluated against the same standards as statistical models - making it again necessary to create an annotated gold standard.

{% hint style="info" %}
Rule-based models can be combined with statistical models to improve performance. Different architectures are possible: You can use a rule-based model and a statistical model simultaneously to optimize for recall. You can also use a broad rule-based model to generate candidates and a statistical model for disambiguation.
{% endhint %}

### Semantic Similarity

Semantic similarity is a method to compare the similarity of two or more texts or documents. It doesn't necessarily represent its own class of approaches, as embeddings can already be used to compare documents. However, there are some models, such as Siamese neural networks, that can be trained for similarity.&#x20;

This can be helpful for many tasks in OJA analysis, such as identifying duplicates, entity linking, and classification.


# Evaluation and Quality Control

Evaluating enrichment and extraction models.

## Evaluation

Model evaluation is a crucial part of developing models for OJA enrichments because it helps you understand how well your model is performing and whether it is meeting the goals that you have set for it. To evaluate machine learning and extraction models, we need a gold standard \
([#gold-standard-annotation-and-quality](#gold-standard-annotation-and-quality "mention")) that experts curated. Against the gold standard, we can calculate different metrics to evaluate the model performance ([#evaluation-metrics](#evaluation-metrics "mention")).

{% hint style="info" %}
Model evaluation for OJA analysis is often more complex than it is for other use cases. This has to do with a) the complexity of the tasks, b) the ambiguity often found in OJA taxonomies, and c) the number of classes we want to predict.

While there are some standard metrics like precision and recall, they often don't tell the whole story and are, on their own, not enough to interpret and judge the performance of a model.
{% endhint %}

### Evaluation Metrics

Which evaluation metrics you use depends heavily on the data, the purpose, and the context of your models. In general, the first step is to calculate a confusion matrix where predicted values and actual values are compared.

The following is an example of a confusion matrix for a classification algorithm that solves the task of classifying zones in an OJA. Each cell is the number of examples for a specific actual vs. predicted value.&#x20;

<figure><img src="/files/L3z188MHCkCg5HZhztZM" alt="Confusion matrix from Gnehm and Clematide (2020) on a study on text zoning an classification."><figcaption><p>Confusion matrix from Gnehm and Clematide (2020) on a study on text zoning and classification.</p></figcaption></figure>

From the confusion table, you would usually calculate the number of True Positives, False Positives, and False Negatives, which can then be used to calculate the following metrics:

1. **Accuracy:** This is the most straightforward metric and measures the proportion of correct predictions. However, it can be misleading when the distribution of classes is not balanced.
2. **Precision**: This is a measure of the false positive rate and is defined as the number of true positives divided by the sum of the true positives and false positives. Intuitively, it represents the expectation "How many cases that were found were actually correct?"
3. **Recall**: This is a measure of the false negative rate and is defined as the number of true positives divided by the sum of the true positives and false negatives. Intuitively, it represents the expectation "How many cases that actually exist were correctly found?"
4. **F1 score** (or F-Score): This is the harmonic mean of precision and recall and is defined as 2 \* (precision \* recall) / (precision + recall).
5. **Confusion matrix**: This table is used to visualize the performance of a classifier by showing the number of true positives, true negatives, false positives, and false negatives.
6. **ROC curve**: This is a plot that shows the true positive rate on the y-axis and the false positive rate on the x-axis and is used to visualize the trade-off between these two rates for different classification thresholds.

Usually, **you would expect that precision and recall are reported**. When these metrics are used, it is essential to distinguish between micro- and macro-scores. The micro-score tells you the performance over all predictions, while the macro-score is the average of the predictive performance by each class or label. Whenever possible, both scores should be calculated and reported.

{% hint style="info" %}
It is hard to specify cut-off points for these metrics in order to judge a model. What constitutes a good result is dependent on many different factors, most importantly the complexity of a task. Some tasks might be semantically easier to model, e.g., finding job titles in a job posting, and you would expect high scores on most metrics (for example .8 + on the F1-Score). On the other hand, more complex tasks, e.g., classifying skills from an 8,000-concept-strong taxonomy, might be considered good if the Macro-F1-Score is above .4.

Furthermore, it is difficult to compare different model implementations that are tested on different gold standard data sets. Therefore, it can be argued that it is the responsibility of the authors of a study or model to critically discuss the performance of the models using benchmarks, examples, or comparison values.
{% endhint %}

### Reporting Results

Reporting results - especially for machine learning models - is an important part of the analysis cycle. It is important to communicate the following points so that others can better interpret and use the results:

* **Dataset:** Provide information about the dataset used. This should include details about its size, nature, and source. Highlight if any pre-processing steps (like cleaning, normalization, feature extraction, etc.) were undertaken.
* **Taxonomy**: Explain which taxonomy was used and why. If the taxonomy was enriched indicate which assumptions were applied and how they influence the results. See more about data standards and metrics in the section [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#data-standards-for-taxonomies-and-ontologies).
* **Methods:** Explain the methodology adopted. Describe the model used, its configuration, and why it was chosen over other possible models. Include details about the algorithm, and the training and validation process.
* **Performance Metrics:** Present and explain the metrics used to evaluate the model's performance (accuracy, precision, recall, F1-score, etc.). Detail how these metrics were calculated, why they were chosen, and what they indicate about the model's performance. An overview of metrics can be found above [#evaluation-metrics](#evaluation-metrics "mention").
* **Results:** Present the results obtained from the model. This can include predictive accuracy, feature importance, confusion matrices, or other relevant findings. Make sure to clarify what these results imply in the context of the study.
* **Limitations:** Discuss the limitations of your study. This might encompass constraints with the data, model, computational resources, or the inherent complexity of the problem.
* **Bias:** Address any potential biases present in the data or the model. This could be in data collection, data representation, or the model's bias-variance trade-off. Acknowledging bias helps in assessing the fairness and generalizability of the model.

A helpful tool for reporting models is the so-called **model card**. Huggingface provides a good overview of the [literature regarding scorecards](https://huggingface.co/docs/hub/model-card-landscape-analysis): The presented tools and processes can be "contextualized with regard to their focus (e.g., on which part of the ML system lifecycle does the tool focus?) and their intended audiences (e.g., who is the tool designed for?)".

An example for a model score card is the [documentation of the extraction algorithm for activity fields ("Teilqualifikationen")](https://www.and-effect.com/white-paper) by the Bertelsmann Stiftung and \&effect data solutions (Müller 2023).

### Gold Standard Annotation and Quality

A gold standard is a data set of documents for which human annotators have added labels. This kind of dataset is usually used to train or fine-tune machine learning models and evaluate them.

The development of a gold standard dataset starts with a taxonomy and documents. Depending on the model purpose, the documents are annotated by experts or trained individuals who assign labels to the documents - on the document level or the token level.&#x20;

To ensure high data quality, annotation guidelines (or coding manuals) are formulated, which help annotators decide on ambiguous cases and explain the concepts to be annotated in more detail.

{% hint style="info" %}
Developing a gold standard is an iterative process. Usually, it is a good idea to introduce the concepts to all annotators. Then you can start by assigning a limited amount of documents to all annotators and measure the disagreements using Fleiss' Kappa. When all annotators are comfortable with the guidelines and the agreement is high, you can still use a document overlap, for example, 20 percent, to assess the quality continuously.
{% endhint %}

Different metrics can be used to measure the annotation quality, such as Fleiss' Kappa or Krippendorff's Alpha ([McHugh 2012](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900052/)). These metrics should be used continuously to assess whether the annotation guidelines are clear enough or which cases produce the most annotation disagreements.&#x20;

{% hint style="info" %}
As with the model evaluation metrics, the quality of an annotated dataset can not only be quantified by annotation agreement metrics. Often used cut-off points for high agreement scores were formulated by [Landis and Koch (1977)](https://www.jstor.org/stable/2529310) as .61 to .80 as substantial agreement and .81 to 1.00 as almost perfect agreement.
{% endhint %}

After finishing annotating a gold standard, usually, you will have some documents with disagreements between annotators. Depending on the size of your dataset, you can either discard the co-annotated documents or adjudicate the dataset. Adjudication is time-intensive but leads to an unambiguous and high-quality gold standard.

{% hint style="info" %}
There is no silver bullet method to calculate how many documents are needed for a gold standard - among other considerations it depends on&#x20;

**1) how semantically complex the task is**: A skill in a job posting can have many different representations in a job posting; whether or not you need a university degree for a position is linguistically far less complex to detect.

**2) how many classes need to be predicted**: A binary classifier (e.g. remote job or not) is far less complex to model and evaluate than a classifier with over 1000 categories (e.g., a skill taxonomy).

**3) how classes are distributed**: Often classes or categories are not distributed evenly in job postings. If a concept (e.g., a skill) appears only in 1 out of 1000 documents, you would need a large multiple of annotated documents to robustly estimate how well your model can extract it.
{% endhint %}


# Taxonomies and Ontologies

Overview of taxonomies and ontologies and their application in OJA analysis.

Taxonomies and ontologies are useful tools for online job ad analysis as they structure and systematize occupations, skills, and other concepts. In the life cycle of an OJA, they provide the crucial link between OJAs and the domain at large.&#x20;

A taxonomy is a hierarchical system for organizing and categorizing concepts.&#x20;

### Taxonomies for Online Job Ad Analysis

For labor market analysis, various taxonomies were developed for different concepts. Expert commissions developed the taxonomies below and update them regularly - making them a good starting point for any OJA-related project. A good starting point is the following list of taxonomies:

<table><thead><tr><th width="161">Indicator / Area</th><th width="133">Name</th><th>Description</th></tr></thead><tbody><tr><td><strong>Occupations</strong></td><td><a href="https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:International_standard_classification_of_occupations_(ISCO)">ISCO</a></td><td>The International standard classification of Occupations is an international classification developed and maintained by the International Labour Organization (ILO) for organizing jobs into a clearly defined set of groups according to the tasks and duties in the job.</td></tr><tr><td></td><td><a href="https://statistik.arbeitsagentur.de/DE/Navigation/Grundlagen/Klassifikationen/Klassifikation-der-Berufe/KldB2010-Fassung2020/KldB2010-Fassung2020-Nav.html">KldB2010</a></td><td>The classification of Occupations (Klassifikation der Berufe 2010; KldB 2010) is the standard occupation taxonomy in Germany. The classification was developed to describe and systematize the occupational landscape in Germany. The KLDB-2010 is a hierarchical classification with five levels. In doing so, occupations are classified according to a horizontal dimension (occupational expertise) and a vertical dimension (requirement level). The Bundesagentur für Arbeit provides a provisional <a href="https://statistik.arbeitsagentur.de/DE/Navigation/Grundlagen/Klassifikationen/Klassifikation-der-Berufe/KldB2010-Fassung2020/Arbeitsmittel/Arbeitsmittel-Nav.html;jsessionid=04DE235EFF3992D5EDBDA3991886403C">mapping</a> of the KldB2010 to ISCO.</td></tr><tr><td></td><td><a href="https://www.onetcenter.org/taxonomy.html">O*NET</a>-SOC</td><td>The O*NET-SOC database contains hundreds of standardized and occupation-specific descriptors on almost 1,000 occupations covering the U.S. labor market. Furthermore, it links knowledge, skills, and abilities to standardized occupations.</td></tr><tr><td><strong>Skills / Competences</strong></td><td><a href="https://esco.ec.europa.eu/en/classification">ESCO</a> (also occupations)</td><td>ESCO is the multilingual classification of European Skills, Competences, Qualifications, and Occupations. It identifies and categorizes the relevant concepts for the EU labor market and education and training in 25 European languages. It uses hierarchical relationships between them, metadata, and mappings to the International Standard Classification of Occupations (ISCO) to structure the occupations.</td></tr><tr><td></td><td><a href="https://web.arbeitsagentur.de/berufenet/">Berufenet</a></td><td>Berufenet is an offering by the Bundesagentur für Arbeit and gives comprehensive information about professions in Germany. With it comes a competence classification scheme.</td></tr><tr><td></td><td><a href="https://www.onetcenter.org/taxonomy.html">O*NET</a></td><td>The Occupational Information Network (O*NET) is developed under the sponsorship of the U.S. Department of Labor/Employment and Training Administration. Apart from the SOC occupations it also provides a comprehensive skill, ability and competence taxonomy.</td></tr><tr><td><strong>Sectors / Industries</strong></td><td><a href="https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:Statistical_classification_of_economic_activities_in_the_European_Community_(NACE)#:~:text=NACE%20is%20a%20four%2Ddigit,developed%20within%20the%20European%20statistical">NACE</a></td><td>The Statistical Classification of Economic Activities in the European Community is a system for classifying economic sectors. The more common classification <a href="https://statistik.arbeitsagentur.de/DE/Navigation/Grundlagen/Klassifikationen/Klassifikation-der-Wirtschaftszweige/Klassifikation-der-Wirtschaftszweige-Nav.html">WZ-2008</a> (Klassifikation der Wirtschaftszweige) is built upon NACE. An economic sector or industry is usually defined as a grouping of similar companies or businesses in terms of the economic activity performed, the manufacturing process, or the products manufactured.</td></tr><tr><td><strong>Regions</strong></td><td><a href="https://ec.europa.eu/eurostat/web/nuts/background">NUTS regions</a></td><td>The nomenclature of territorial units for statistics is a geographical system, according to which the territory of the European Union is divided into hierarchical levels. The three hierarchical levels are known as NUTS-1, NUTS-2, and NUTS-3. This classification enables cross-border statistical comparisons at various regional levels within the EU.</td></tr><tr><td></td><td><a href="https://www.iab-forum.de/glossar/arbeitsmarktregionen/">Labor Market Regions (DE: Arbeitsmarktregionen)</a></td><td>Labor market regions offer the advantage of reflecting the spatial aspects of economic activities as accurately as possible, thus providing relevant analytical units for regional research.</td></tr><tr><td><strong>Education / Certificates</strong> </td><td><a href="https://ec.europa.eu/eurostat/statistics-explained/index.php?title=International_Standard_Classification_of_Education_(ISCED)">ISCED</a></td><td>The International Standard Classification of Education (ISCED) is the reference international classification for organizing educational programs and related qualifications by levels and fields.</td></tr><tr><td><strong>Applicant Attraction</strong></td><td><a href="https://jade.or.at">Job Ad Decoder Dictionary</a></td><td>A classification of employer attributes, tasks, candidate requirements, and offered benefits on the basis of applicant preferences in regard to gender, age (groups), and (as of January 2024) cultural background. The classification was developed at the University of Innsbruck (Austria) based on representative surveys with German-speaking labor market participants (potential job applicants). In its current form, the Job Ad Decoder Dictionary (JADE Dictionary) contains 468 job ad-related words most of which were found to appear more attractive to men than to women or vice versa, more attractive to younger applicants than to older applicants or vice versa, or to be relevant in both diversity dimensions. The dictionary is directly being used to decode bias in job ads in the online tool Job Ad Decoder.</td></tr></tbody></table>

{% hint style="info" %}
&#x20;There are two main advantages of using standardized taxonomies. Firstly, it ensures that models build on them and analyses derived from those models are interoperable and relevant to stakeholders at different levels. Using the KldB-2010 taxonomy, for example, ensures that you can cross-reference your findings and analysis with official analysis by the Federal Statistics Office or the Bundesagentur für Arbeit. Secondly, the taxonomies were developed by experts and are relevant to the labor market.

The main challenge of using these pre-defined taxonomies is that they were developed for many use cases and might not fit the problem you want to model directly.

One way of mitigating this challenge is to develop a custom taxonomy for your problem while ensuring you can link its concepts to them.
{% endhint %}

### Developing a Taxonomy

{% embed url="<https://youtu.be/vtrKAde4Wi4>" %}
Presentation by Prof. Dr. Matthias Ziegler at the OJV Forum 2021 about developing ontologies for OJA analysis.
{% endembed %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-gewinnung-von-taxonomien/>" %}

Developing a taxonomy for a data project involves organizing and categorizing data in a logical and meaningful way. There are several approaches you can take when developing a taxonomy. Three general ways were defined by [Uschold and Gruninger (1996)](https://www.cambridge.org/core/journals/knowledge-engineering-review/article/abs/ontologies-principles-methods-and-applications/2443E0A8E5D81A144D8C611EF20043E6):

1. **Bottom-up**: This approach starts with the most granular or specific data points and then groups them into larger, more general categories. For example, you can start by identifying individual job titles, then group those titles into broader job functions (such as "engineering" or "sales"), and then group those functions into even broader industry categories. For this approach [Extraction Methods](/steps/extraction-methods#unsupervised-classification) can be used to develop a data-driven taxonomy.
2. **Top-down**: This approach involves starting with the most general categories and then breaking them down into smaller, more specific subcategories. For example, you can start with broad industry categories and then drill down into specific job functions and individual job titles.
3. **Middle-out**: This approach involves starting with a general set of categories and then adding additional layers of specificity as needed. This approach can be helpful when you have a large amount of data and you're not sure where to start.

Regardless of which approach you choose, it's important to ensure that your taxonomy is logical, intuitive, and most importantly fits your use case.&#x20;

> \[...] some fundamental rules in ontology design \[...]. These rules may seem rather dogmatic. They can help, however, to make design decisions in many cases.
>
> 1\) There is no one correct way to model a domain— there are always viable alternatives. The best solution almost always depends on the application that you have in mind and the extensions that you anticipate. \
> 2\) Ontology development is necessarily an iterative process. \
> 3\) Concepts in the ontology should be close to objects (physical or logical) and relationships in your domain of interest. These are most likely to be nouns (objects) or verbs (relationships) in sentences that describe your domain.
>
> [Ontology Development 101: A Guide to Creating Your First Ontology](https://protege.stanford.edu/publications/ontology_development/ontology101.pdf)

{% embed url="<https://blog.aus-und-weiterbildung.eu/employees-wanted-analysis-of-online-job-advertisements-a-series-4-6/>" %}

### Data Standards for Taxonomies and Ontologies

Semantic Modeling is a difficult endeavor with many semantic and logical pitfalls and trade-offs. Kai Krüger (BIBB) gives the following example:

> When it comes to semantic modelling different approaches yield different results. This must be taken into account when interpreting the results. Perhaps it becomes clearest with an (exaggerated) example:
>
> In a job description, is a sentence like:&#x20;
>
> `"[...] Tasks: Occasional support in research activities on new procedures in the field of Machine Learning"`
>
> Now *Machine Learning* (as a technical skill) could be interpreted as part of the person's tasks or the emphasis is placed on the *professional research* (and even then it's only supporting and only partially). Both interpretations have advantages and disadvantages.&#x20;
>
> One could discuss a number of points here that fall under the heading of semantic modeling. However, if you only publish "our model with an F1 score of 97% concludes that tasks in the field of AI have increased by 10%", then all these points are lost. Therefore, things like the structure or selection of a taxonomy or the development of annotation guidelines are not only tasks that are important to achieve the result, but both reflecting on this process and making it transparent are essential prerequisites for the epistemological value, as well as the connectability and comparability of the study.
>
> *Kai Krüger (BIBB)*

For the evaluation of an ontology - especially in the context of an information extraction task - one must consider both the ontology itself and the results of the extraction. For this, Panos Alexopolous (2020) formulates the following [dimensions](https://www.oreilly.com/library/view/semantic-modeling-for/9781492054269/ch04.html) in his book on semantic modelling.&#x20;

* **Semantic Precision** (= “the degree to which the semantic assertions of a model are accepted to be true”): How precise is the extraction of entities given the ontology and extraction rules? The precision of the extraction is influenced by various aspects: errors within the taxonomy (e.g., entities are not clearly described in the taxonomy), errors in the gold standard (e.g., employers provide the wrong entity for a job posting), lack of expert knowledge from the involved parties (e.g., keywords are incorrectly assigned), vagueness and ambiguity in the taxonomy (e.g., the assignment of a job title is not clear and different experts would make different decisions).&#x20;
* **Completeness** (= “the degree to which elements that should be contained in the model are indeed there”): How many of the actual entities can be found? The completeness of the extraction is primarily influenced by the completeness of the taxonomy and ontology. The more search terms used for extraction, the higher the completeness. Completeness is calculated via the recall of the extraction.&#x20;
* **Consistency** (= “a semantic model is free of logical or semantic contradictions”): The consistency of the ontology is determined by how consistent the enrichment process is.
* **Ambiguity** (\~conciseness = “the degree to which the model does not contain redundant elements”): The succinctness and vice versa the ambiguity of the ontological extraction can be measured at two levels: At the ontology level, it can be determined by the number of search terms that are assigned to multiple entities (Ontological Ambiguity). At the extraction level (Extraction Ambiguity), it can be measured how many search terms each identify 1) only True Positives (no ambiguity), 2) both True Positives and False Positives, and 3) multiple False Positives.&#x20;
* **Timeliness** (= “the degree to which the model contains elements that reflect the current version of the world”): The currency of the taxonomy is determined by the currency of the entities and all additional attributes, relations, and search words. Professions and competencies and their relations change over time. This primarily results in challenges regarding the maintenance of the models.&#x20;
* **Relevancy** (= “model is relevant when its structure and content are useful and important for a given task or application“): The relevance of the model largely depends on the local study context and specific application.&#x20;
* **Understandability** (= “the ease with which human consumers can understand and utilize the model’s elements, without misunderstanding or doubting their meaning“): The understandability of the extraction particularly refers to how comprehensible the extraction is and whether the results are plausible.&#x20;
* **Trustworthiness** (= "the perception and confidence in the quality of the model by its users")

These dimensions often conflict with each other. Completeness and precision, for example, must be considered together: the more unspecific search terms one includes in the model, the higher the likelihood of recognising entities, but precision decreases at the same time. There's a similar trade-off between completeness and low ambiguity: specific search terms can fundamentally describe several entities. By minimizing the number of unspecific search terms, the completeness of the extraction also decreases.


# Dataset Curation and Representativity Analysis

After collecting, structuring, and enriching OJA data, you usually want to create a sample for analysis. At this stage, there are still duplicates in the dataset, and depending on your data collection strategy, there might be structural bias.&#x20;

### Deduplication

We have already dealt with [Data Enrichment](/steps/data-enrichment#identifying-duplicates) as part of the data enrichment phase. However, removing duplicates for analysis warrants its own section, as there are trade-offs to be aware of.

Deduplicating online job ads (after a duplicate identification algorithm has marked duplicates) can be done in several ways, depending on the specific requirements of your project:

1. **Automatic removal**: One approach is to remove all duplicates identified by the algorithm automatically. This can be done by simply deleting duplicate records from the dataset on some property. Usually, this is the publishing date - where you only keep the earliest record of this particular job. Alternatively, job postings can also be picked at random from all instances. \
   The automatic removal can lead to the loss of important information: If job postings are collected from different job portals, the parsing might work better on one platform than another, and you would lose indicators that can be extracted from one but not the other.
2. **Custom rules**: Another approach is manually setting more intricate rules for deduplication. For example, if a job posting for a vacancy is available from one website with high data quality, take this one; otherwise, take the job posting from another portal. This approach, however, might introduce another kind of bias to the dataset, which you should analyze and be aware of.
3. **Merging**: A third approach is to merge the duplicates into a single record, typically by starting with the earliest job posting and then merging any additional information or indicators available in other instances. This can help to retain all relevant information but can also introduce inaccuracies when there is conflicting information in different postings. Furthermore, this is computationally the most expensive form of deduplicating.

When deduplicating job ads, there are several trade-offs to consider. Automatic removal is the quickest and easiest approach but can lead to losing important information. Merging the duplicates can be helpful if you want to retain all relevant information, but can lead to the loss of important information. Ultimately, it depends on the project's goal and the indicators' distribution.

{% hint style="info" %}
Automatic removal is usually a good strategy as it distributes indicator values randomly. This is provided there is no bias in the property you are removing duplicates on. This approach doesn't change the distributions of the indicators and doesn't introduce any new bias to the data.
{% endhint %}

### Filtering Data

When creating a dataset for analysis, one thing to consider is whether or not to remove certain job ads from the analysis. For example, job postings published by temporary work or recruitment agencies. These agencies often search broadly for candidates who can be staffed to fill vacancies at different companies. Other job postings that could be excluded from the analysis are internship, volunteer, or freelancer positions.

### Aggregation and Unit of Observation

Before analyzing trends or making comparisons, OJA data is often aggregated. Here are some of the aggregations usually made:

* **Aggregation by region**: The NUTS regions are typical aggregation levels for regional analysis. The Nomenclature of Territorial Units (NUTS) classification of the European Union is a hierarchical system that defines the socio-economic regions of the European Union at different levels. The hierarchical arrangement of the NUTS classification allows data to be aggregated across regions - for example, on level 1 in Germany, the states (Bundesländer), and on level 3, the districts (Landkreise). A prerequisite for this aggregation is usually that the location or the geo-location of the vacancy is provided in the metadata.&#x20;
* **Aggregation by time**: OJAs can be grouped by day, week, month, quarter, or year based on the information provided when a job ad was published. Analyzing daily new postings can be challenging, with high volatility and stark inter-day variance. Grouping data in bigger time intervals, such as months or quarters, makes analysis usually more robust. Another way of making time-series analysis more robust is not looking at the new posting but instead at the stock of active vacancies/postings. However, in practice, you need a publishing date and when the posting was retired or the vacancy filled (which often is not available). The Statistical Office used a "[Pseudostock](https://www.destatis.de/DE/Methoden/WISTA-Wirtschaft-und-Statistik/2021/03/auswirkungen-coronakrise-arbeitsmarkt-032021.html)" method to estimate the current stock of active job ads ([Lazzar and Rengers 2021](https://www.destatis.de/DE/Methoden/WISTA-Wirtschaft-und-Statistik/2021/03/auswirkungen-coronakrise-arbeitsmarkt-032021.html)).&#x20;
* **Aggregation by indicators**: Lastly, data can be grouped on any extracted property. Most common would be industry (e.g., Klassifikation der Wirtschaftszweige 2008), occupation (e.g., Klassifikation der Berufe 2010), skills (e.g., ESCO), education level (e.g., ISCED), salary ranges, or employment type. For more taxonomies, see [Taxonomies and Ontologies](/steps/taxonomies-and-ontologies#taxonomies-for-online-job-ad-analysis).

{% hint style="info" %}
The level of aggregation is an important one for analysis. Especially when making comparisons, it is essential to remember that the more combinations of indicators you analyze, the smaller the observations in each group become. For example, even when you are starting with 10 million online job vacancies per year, grouping them by month (12 categories), district in Germany (400 categories), and by occupation (\~1,250 categories) gives you already 6 million groups which make some statistical analysis meaningless.&#x20;
{% endhint %}

### Representativity Analysis

Representativity analysis closes the circle to the beginning of the OJA cycle: The question of how and where job postings are published in the first place. And consequently, are OJAs representative of the labor market at large?

Here is an example of the checks that Cedefop proposes in their 2022 paper:&#x20;

> "Two external data sources, the Labour force survey (LFS) and the Job vacancies survey (JVS), were used to evaluate the selectivity of data in terms of several criteria (e.g. comparisons on sectoral, occupational and geographic levels). The comparison of occupations listed in the European skills, competences and occupations classification (ESCO) ( 3 ) taxonomy was used to identify occupations on the labour market absent from OJAs." ([Cedefop 2022](https://www.cedefop.europa.eu/files/6217_en.pdf))

{% hint style="info" %}
The type of representativity analysis that needs to be done for analysis depends mainly on the inference you want to provide. When testing hypotheses about the general labor market space, one must be more careful than with hypotheses about the online job ad space, for example.&#x20;
{% endhint %}

{% embed url="<https://youtu.be/z5OCGFYnKj8>" %}

{% embed url="<https://blog.aus-und-weiterbildung.eu/oja-sicherung-der-qualitaet/>" %}


# Literature and Projects

Literature overview and reference

## Literature

Binnewitt, J., & Schnepf, T. (2021). Wortliste zur Assoziation ökologischer Nachhaltigkeit. PUF\_1.0. Bonn: Bundesinstitut für Berufsbildung. [doi:10.7803/550.21.1.7.10](https://doi.org/10.7803/550.21.1.7.10)

Buchmann, M., Buchs, H., Busch, F., Clematide, S., Gnehm, A.-S., & Müller, J. (2022). Swiss Job Market Monitor: A Rich Source of Demand-Side Micro Data of the Labour Market. *European Sociological Review, jcac002*. doi: 10.1093/esr/jcac002

Büchel, J., Mertens, A., 2021, KI-Bedarfe der Wirtschaft am Standort Deutschland. Eine Analyse von Stellenanzeigen für KI-Berufe. *Gutachten im Auftrag des Bundesministeriums für Wirtschaft und Energie, Köln*

Cedefop. (2018). Mapping the landscape of online job vacancies: Background country report: Germany. [cedefop.europa.eu](http://www.cedefop.europa.eu/en/events-and-projects/projects/big-data-analysis-onlinevacancies/publications)

Cedefop. (2019). Online job vacancies and skills analysis: A Cedefop pan-European approach. Publications Office. [doi:10.2801/097022](https://data.europa.eu/doi/10.2801/097022)

De Lazzer, J., & Rengers, M. (2021). Auswirkungen der Coronakrise auf den Arbeitsmarkt: Experimentelle Statistiken aus Daten von Online-Jobportalen. *WiSta, 3/2021*. Retrieved from <https://www.destatis.de/DE/Methoden/WISTA-Wirtschaft-und-Statistik/2021/03/auswirkungen-coronakrise-arbeitsmarkt-032021.pdf?__blob=publicationFile>

Eggenhofer-Rehart, P., Brandl, J., & Kohlberger, M. (2022). ‚Flexibility and flexibility are not the same’: A genre-sensitive method to measuring gendered wording in job advertisements. Vortrag am Herbstworkshop der Wissenschaftlichen Kommission Personalwesen im Verband der Hochschullehrer für Betriebswirtschaft e.V., Berlin, Germany, 29-30 September 2022

Gnehm, A.-S., & Clematide, S. (2020). Text Zoning and Classification for Job Advertisements in German, French and English. In *Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science* (pp. 83-93). Association for Computational Linguistics.

Gnehm, A.-S., Bühlmann, E., & Clematide, S. (2022). Evaluation of Transfer Learning and Domain Adaptation for Analyzing German-Speaking Job Advertisements. In Proceedings of the 13th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association.

Grimm, V., Janser, M., & Stops, M. (2021). Neue Analyse von Online-Stellenanzeigen: Kompetenzen für die Wasserstofftechnologie sind jetzt schon gefragt. *IAB-Kurzbericht, 11/2021*. Nürnberg.

Hermes, J., & Schandock, M. (2016). Stellenanzeigenanalyse in der Qualifikationsentwicklungs-forschung. Die Nutzung maschineller Lernverfahren zur Klassifikation von Textabschnitten. Bonn.

Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159–174. <https://doi.org/10.2307/2529310>

McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 22(3), 276-82. PMID: 23092060; PMCID: PMC3900052.

Murauer, B., Tschuggnall, M., Specht, G., & Brandl, J. (2023). Algorithmic Segmentation of Job Ads Using Textual Analysis. In Computational Linguistics and Intelligent Text Processing: 19th International Conference, CICLing 2018, Hanoi, Vietnam, March 18–24, 2018, Revised Selected Papers, Part II. Springer-Verlag, Berlin, Heidelberg, 287–300. <https://doi.org/10.1007/978-3-031-23804-8_23>

Müller, J., Fingerhut, J., & Noack, M. (2022). Algorithmische Extraktion transversaler Kompetenzen nach ESCO v1.1 aus deutschsprachigen Online-Jobanzeigen. <https://www.and-effect.com/white-paper>

Müller, J. (2023). Extraction of activity fields from German online job posting – Model Card and Dataset Card. <https://www.and-effect.com/white-paper>

Napierala, J., Kvetan, V., & Branka, J. (2022). Assessing the representativeness of online job advertisements. Luxembourg: Publications Office. Cedefop working paper, No 17. <http://data.europa.eu/doi/10.2801/807500>

Noack, M., Ziegler, M., & Müller, J. (2022). Kompetenzwandel in Krisenzeiten – Welche Soft Skills jetzt zählen. Gütersloh: Bertelsmann Stiftung. <https://www.bertelsmann-stiftung.de/de/publikationen/publikation/did/kompetenzwandel-in-krisenzeiten-all-1>

Noack, M. & Müller, J. (2023). Mehr Fachkräfte in jeder Region - Deckung des regionalen Fachkräftebedarfs durch Teilqualifikationen. Gütersloh: Bertelsmann Stiftung.

Noy, N. F., & McGuinness, D. L. (2001). Ontology development 101: A guide to creating your first ontology.

Panos, A. (2020): [Semantic modeling for data. Avoiding pitfalls and breaking dilemmas.](https://www.oreilly.com/library/view/semantic-modeling-for/9781492054269/) O'Reilly Media.

Plaimauer, C. (2016). Big Data als Informationsquelle für regionales Arbeitsmarkt-Monitoring: Online-Stellenanzeigen analysieren mittels Jobfeed. AMS info, 339.

Plaimauer, C. (2018). Die Darstellung der Arbeitskräfte-Nachfrage im AMS-Qualifikations-Barometer. AMS info, 410/411.&#x20;

Plaimauer, C. (2020). Using big data analysis to develop labour market taxonomies: The case of the Austrian Public Employment Services skills taxonomy. In The feasibility of using big data in anticipating and matching skills needs (pp. 36-43). ISBN: 978-92-2-032854-5.

Uschold, M., & Gruninger, M. (1996). Ontologies: Principles, methods and applications. The Knowledge Engineering Review, 11(2), 93-136. doi:10.1017/S0269888900007797

Rengers, M. (2020). Developing meaningful indicators based on OJA \[online job vacancies] data. In ESSnet Big Data, Work Package B: Implementation – Online Job Vacancies. [ec.europa.eu](https://ec.europa.eu/eurostat/cros/sites/default/files/DE_OJA-meaningful-indicators_2020_08_24.pdf)

Rengers, M. (2018). Internetgestützte Erfassung offener Stellen – Machbarkeitsstudie im Rahmen eines ESSnet-Projekts zu Big Data. WiSta, 5, 11ff. [destatis.de](https://www.destatis.de/DE/Methoden/WISTA-Wirtschaft-und-Statistik/2018/05/internetgestuetzte-erfassung-052018.pdf?__blob=publicationFile\&v=4)

Rengers, M. (2018). Internetbasierte Erfassung offener Stellen im Statistischen Bundesamt. Schriftenreihe der ASI – Arbeitsgemeinschaft Sozialwissenschaftlicher Institute, Wiesbaden.

Stephany, F. (2021). One size does not fit all: Constructing complementary digital reskilling strategies using online labour market data. Big Data & Society. <https://doi.org/10.1177/20539517211003120>\
\
Stephany, F. (2020) When Does it Pay Off to Learn a New Skill? Revealing the Complementary Benefit of Cross-Skilling. [http://dx.doi.org/10.2139/ssrn.3717077](https://dx.doi.org/10.2139/ssrn.3717077)

Stops, M. (2021). Kompetenz-Kompass: Mit einem neuen Verfahren lassen sich die Kompetenzanforderungen in Stellenanzeigen systematisch abbilden. *IAB-Forum*. Retrieved from [https://www.iab-forum.de](https://www.iab-forum.de/)

Stops, M., Bächmann, A.-C., Glassner, R., Janser, M., Matthes, B., Metzger, L.-J., Müller, C., & Seitz, J. (2021). Extracting Skill Requirements from Job Ads – the 'Machbarkeitsstudie Kompetenz-Kompass'. *IAB-Forschungsbericht, 07/2021*. Nürnberg.

Stops, M., Bächmann, A.-C., Glassner, R., Janser, M., Matthes, B., Metzger, L.-J., Müller, C., Seitz, J., Hanebrink, A. (Mitarb.), Lataster, L. (Mitarb.), & Müller, G. (Mitarb.). (2020). Machbarkeitsstudie Kompetenz-Kompass \* Teilprojekt 2: Beobachtung von Kompetenzanforderungen in Stellenangeboten. *Bundesministerium für Arbeit und Soziales. Forschungsbericht, 553*. Berlin.

Stops, M., Laible, M.-C., & Leschnig, L. (2022). Überfachliche Kompetenzen sind gefragt – allen voran Zuverlässigkeit und Teamfähigkeit. *IAB-Forum*. Retrieved from [https://www.iab-forum.de](https://www.iab-forum.de/)

Vogel, P., & Hilgendorf, E. (2018). Web Scraping in der unabhängigen wissenschaftlichen Forschung - Gutachten im Auftrag des Wissenschaftszentrums Berlin für Sozialforschung gGmbH (WZB) – Rat für Sozial- und Wirtschaftsdaten (RatSWD).

Zhao, Y., Chen, H., & Mason, C. M. (2022). A Framework for Duplicate Detection from Online Job Postings. In IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT '21). Association for Computing Machinery, New York, NY, USA, 249–256. <https://doi.org/10.1145/3486622.3493928>

## Projects

{% hint style="success" %}
This section is still in progress. If you have any projects you want to highlight here, please write <johannes.mueller@and-effect.com>
{% endhint %}

Suisse Stellenmarkt-Monitor: <https://www.stellenmarktmonitor.uzh.ch/de.html>

Nesta's Open Jobs Observatory: <https://www.nesta.org.uk/data-visualisation-and-interactive/open-jobs-observatory/>&#x20;

Jobmonitor der Bertelsmann Stiftung: <https://jobmonitor.de/>\
\
Job Ad Decoder: <http://www.jade.or.at/>