transformers

Currently Crawler does not let JavaScript run on crawled pages. This leads to issues for websites that rely on fully dynamically loaded content that is not present in the DOM at loading time, but is fetched with JS immediately after.

We should investigate how to allow JavaScript to run on the crawled pages with Selenium and add a switch to the Crawler that allows users to do this.

Based on

欢迎您反馈PaddleNLP使用问题，非常感谢您对PaddleNLP的贡献！
在留下您的问题时，辛苦您同步提供如下信息：

• 版本、环境信息
  1）PaddleNLP和PaddlePaddle版本：请提供您的PaddleNLP和PaddlePaddle版本号，例如PaddleNLP 2.0.4，PaddlePaddle2.1.1
  2）系统环境：请您描述系统类型，例如Linux/Windows/MacOS/，python版本
• 复现信息：如为报错，请给出复现环境、复现步骤
  paddle版本2.0.8 paddlenlp版本2.1.0
  建议，能否在paddlenlp文档中，整理列出各个模型的tokenizer是基于什么类别的based，如bert tokenizer是word piece的，xlnet tokenizer是sentence piece的，以及对应的输入输出样例

Describe the bug
When running tools/preprocess_data.py to tokenize my dataset, I was confused why the generated .bin and .idx files were empty. It turns out that lm_dataformat, the library which actually reads the dataset into the tokenization logic, was version 0.0.19 as specified in the requirements.txt file. This version of the library doesn't include support for uncompressed `.jso

Hi there.
The app is not launching on ubuntu 21.10 (kernel 5.13.0-39). When running from terminal I get:

Fontconfig warning: "/etc/fonts/conf.d/10-hinting-slight.conf", line 5: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/conf.d/10-hinting-slight.conf", line 5: invalid 

Hey! Thanks for the work on this.

Wondering how we can use this with mocha? tsconfig-paths has its own tsconfig-paths/register to make this work

https://github.com/dividab/tsconfig-paths#with-mocha-and-ts-node

Basically with mocha we have to run mocha -r ts-node/register -- but that wouldnt have the compiler flag.

Would be worthwhile to have the ability to do it which looks like