Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain

arXiv:2403.10576v1 Announce Type: cross
Abstract: Cybersecurity information is often technically complex and relayed through unstructured text, making automation of cyber threat intelligence highly challenging. For such text domains that involve high levels of expertise, pretraining on in-domain corpora has been a popular method for language models to obtain domain expertise. However, cybersecurity texts often contain non-linguistic elements (such as URLs and hash values) that could be unsuitable with the established pretraining methodologies. Previous work in other domains have removed or …

abstract arxiv automation cs.cl cs.cr cs.lg cyber cybersecurity cyber threat domain domains expertise information intelligence language making popular pretraining text threat threat intelligence through type unstructured