A browsing plugin example with page scrolling.

Browser Plugin Function Example

You are a helpful AI assistant. You may call available functions repeatedly before responding to the user.

Assistant Functions

@pb def

{
  "name": "browse_url",
  "description": "Browse the contents of a website and parse it into markdown. Large websites (> 400 words) will be paginated, use the pagination parameter (section) in subsequent calls to request additional sections. Website metadata and pagination info will be provided in YAML at the top of the output.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The URL of the website to browse"
      },
      "section": {
        "type": "number",
        "description": "Pagination parameter to specify which section of the page to view. Defaults to 0 (the first section)"
      }
    },
    "required": ["url"]
  }
}

Get Current Weather

https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification summarize

Using BERT for Long Text Classification

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\"\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 0
num_sections: 4
---

We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text How can BERT be used?

You have basically three options:

1. You can cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient.
2. You can split your text in multiple subtexts, classify each of them and combine the results back together ( choose the class which was predicted for most of the subtexts for example). This option is obviously more expensive.

How to Use Bert for Long Text Classification?

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 1\n}"
}

pb call

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 1
num_sections: 4
---

3. You can even feed the output token for each subtext (as in option 2) to another network (but you won't be able to fine-tune) as described in [this discussion](https://github.com/google-research/bert/issues/27).

I would suggest to try option 1, and only if this is not good enough to consider the other options.

Using Output Tokens in Neural Networks

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 2\n}"
}

Using BERT for Long Text Classification in Python

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 2
num_sections: 4
---

This paper compared a few different strategies: [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/abs/1905.05583). On the IMDb movie review dataset, they actually found that cutting out the middle of the text (rather than truncating the beginning or the end) worked best! It even outperformed more complex "hierarchical" approaches involving breaking the article into chunks and then recombining the results.

As another anecdote, I applied BERT to the Wikipedia Personal Attacks dataset here, and found that simple truncation worked well enough that I wasn't motivated to try other approaches :)

BERT for Text Classification: Comparing Strategies

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 3\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 3
num_sections: 4
---

You can leverage from the HuggingFace Transformers library that includes the following list of Transformers that work with long texts (more than 512 tokens):

- Reformer: that combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences.
- Longformer: with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer.
- Eight other recently proposed efficient Transformer models include Sparse Transformers (Child et al.,2019), Linformer (Wang et al., 2020), Sinkhorn Transformers (Tay et al., 2020b), Performers (Choromanski et al., 2020b), Synthesizers (Tay et al., 2020a), Linear Transformers (Katharopoulos et al., 2020), and BigBird (Zaheeret al., 2020).

The paper from the authors from Google Research and DeepMind tries to make a comparison between these Transformers based on Long-Range Arena "aggregated metrics":

![Performance, speed, and memory footprint of the Transformers](https://i.stack.imgur.com/TMD6h.png?s=256)

They also suggest that Longformers have better performance than Reformer when it comes to the classification task.

HuggingFace Transformers for long texts

The StackOverflow discussion on how to use BERT for long text classification provides several strategies:

1. Truncate longer texts and only use the first 512 tokens. This is the default behavior of the original BERT implementation and is usually sufficient for most cases.

2. Split the text into multiple subtexts, classify each of them, and then combine the results. This could mean choosing the class predicted most frequently across the subtexts. However, this method is more computationally expensive.

3. Feed the output token for each subtext to another network. This method doesn't allow for fine-tuning and is described in more detail in a [GitHub discussion](https://github.com/google-research/bert/issues/27).

A paper titled [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/abs/1905.05583) compared different strategies and found that removing the middle of the text worked best on the IMDb movie review dataset. This approach even outperformed more complex "hierarchical" methods that involved breaking the article into chunks and recombining the results.

The HuggingFace Transformers library includes several Transformers that work with long texts (more than 512 tokens), such as Reformer and Longformer. These models can process documents of thousands of tokens or longer. Other efficient Transformer models include Sparse Transformers, Linformer, Sinkhorn Transformers, Performers, Synthesizers, Linear Transformers, and BigBird. A comparison of these Transformers based on Long-Range Arena "aggregated metrics" suggests that Longformers perform better than Reformers in classification tasks.

BERT Long Text Classification Strategies

A browsing plugin example with page scrolling.

You are a helpful AI assistant. You may call available functions repeatedly before responding to the user.

@pb def

{
  "name": "browse_url",
  "description": "Browse the contents of a website and parse it into markdown. Large websites (> 400 words) will be paginated, use the pagination parameter (section) in subsequent calls to request additional sections. Website metadata and pagination info will be provided in YAML at the top of the output.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "The URL of the website to browse"
      },
      "section": {
        "type": "number",
        "description": "Pagination parameter to specify which section of the page to view. Defaults to 0 (the first section)"
      }
    },
    "required": ["url"]
  }
}

https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification summarize

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\"\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 0
num_sections: 4
---

We know that BERT has a max length limit of tokens = 512, So if an article has a length of much bigger than 512, such as 10000 tokens in text How can BERT be used?

You have basically three options:

1. You can cut the longer texts off and only use the first 512 Tokens. The original BERT implementation (and probably the others as well) truncates longer sequences automatically. For most cases, this option is sufficient.
2. You can split your text in multiple subtexts, classify each of them and combine the results back together ( choose the class which was predicted for most of the subtexts for example). This option is obviously more expensive.

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 1\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 1
num_sections: 4
---

3. You can even feed the output token for each subtext (as in option 2) to another network (but you won't be able to fine-tune) as described in [this discussion](https://github.com/google-research/bert/issues/27).

I would suggest to try option 1, and only if this is not good enough to consider the other options.

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 2\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 2
num_sections: 4
---

This paper compared a few different strategies: [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/abs/1905.05583). On the IMDb movie review dataset, they actually found that cutting out the middle of the text (rather than truncating the beginning or the end) worked best! It even outperformed more complex "hierarchical" approaches involving breaking the article into chunks and then recombining the results.

As another anecdote, I applied BERT to the Wikipedia Personal Attacks dataset here, and found that simple truncation worked well enough that I wasn't motivated to try other approaches :)

@pb call

{
  "name": "browse_url",
  "arguments": "{\n  \"url\": \"https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification\",\n  \"section\": 3\n}"
}

@pb name=browse_url

---
title: How to use Bert for long text classification?
section: 3
num_sections: 4
---

You can leverage from the HuggingFace Transformers library that includes the following list of Transformers that work with long texts (more than 512 tokens):

- Reformer: that combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences.
- Longformer: with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer.
- Eight other recently proposed efficient Transformer models include Sparse Transformers (Child et al.,2019), Linformer (Wang et al., 2020), Sinkhorn Transformers (Tay et al., 2020b), Performers (Choromanski et al., 2020b), Synthesizers (Tay et al., 2020a), Linear Transformers (Katharopoulos et al., 2020), and BigBird (Zaheeret al., 2020).

The paper from the authors from Google Research and DeepMind tries to make a comparison between these Transformers based on Long-Range Arena "aggregated metrics":

![Performance, speed, and memory footprint of the Transformers](https://i.stack.imgur.com/TMD6h.png?s=256)

They also suggest that Longformers have better performance than Reformer when it comes to the classification task.

The StackOverflow discussion on how to use BERT for long text classification provides several strategies:

1. Truncate longer texts and only use the first 512 tokens. This is the default behavior of the original BERT implementation and is usually sufficient for most cases.

2. Split the text into multiple subtexts, classify each of them, and then combine the results. This could mean choosing the class predicted most frequently across the subtexts. However, this method is more computationally expensive.

3. Feed the output token for each subtext to another network. This method doesn't allow for fine-tuning and is described in more detail in a [GitHub discussion](https://github.com/google-research/bert/issues/27).

A paper titled [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/abs/1905.05583) compared different strategies and found that removing the middle of the text worked best on the IMDb movie review dataset. This approach even outperformed more complex "hierarchical" methods that involved breaking the article into chunks and recombining the results.

The HuggingFace Transformers library includes several Transformers that work with long texts (more than 512 tokens), such as Reformer and Longformer. These models can process documents of thousands of tokens or longer. Other efficient Transformer models include Sparse Transformers, Linformer, Sinkhorn Transformers, Performers, Synthesizers, Linear Transformers, and BigBird. A comparison of these Transformers based on Long-Range Arena "aggregated metrics" suggests that Longformers perform better than Reformers in classification tasks.