Logo: Deutsches Institut für Internationale Pädagogische Forschung

Publications

Publikationendatenbank

show results

Autor:
Remus, Steffen; Biemann, Chris:

Titel:
Domain-specific corpus expansion with focused webcrawling

Quelle:
In: European Language Resources Association (Hrsg.): Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016) Portoroz : European Language Resources Association (2016) , 3607-3611

URL des Volltextes:
http://www.lrec-conf.org/proceedings/lrec2016/pdf/316_Paper.pdf

Sprache:
Englisch

Dokumenttyp:
4. Beiträge in Sammelwerken; Tagungsband/Konferenzbeitrag/Proceedings

Schlagwörter:
Algorithmus, Automatisierung, Bildung, Computerlinguistik, Data Mining, Hypertext, Modell, Sprache, Text, Textanalyse


Abstract(original):
This work presents a straightforward method for extending or creating in-domain web corpora by focused webcrawling. The focused webcrawler uses statistical N-gram language models to estimate the relatedness of documents and weblinks and needs as input only N-grams or plain texts of a predefined domain and seed URLs as starting points. Two experiments demonstrate that our focused crawler is able to stay focused in domain and language. The first experiment shows that the crawler stays in a focused domain, the second experiment demonstrates that language models trained on focused crawls obtain better perplexity scores on in-domain corpora. We distribute the focused crawler as open source software. (DIPF/Orig.)


DIPF-Abteilung:
Informationszentrum Bildung

Notizen:

last modified Nov 11, 2016