Course detail

Natural Language Processing

FIT-ZPDAcad. year: 2024/2025

Foundations of the natural language processing, historical perspective, statistical NLP and modern era dominated by machine learning and, specifically, deep neural networks. Meaning of individual words, lexicology and lexicography, word senses and neural architectures for computing word embeddings, word sense classification and inferrence. Constituency and dependency parsing, syntactic ambiguity, neural dependency parsers. Language modeling and its applications in general architectures. Machine translation, historical perspective on the statistical approach, neural translation and evaluation scores. End-to-end models, attention mechanisms, limits of current seq2seq models. Question answering based on neural models, information extraction components, text understanding challenges, learning by reading and machine comprehension. Text classification and its modern applications, convolutional neural networks for sentence classification. Language-independent representations, non-standard texts from social networks, representing parts of words, subword models. Contextual representations and pretraining for context-dependent language modules. Transformers and self-attention for generative models. Communication agents and natural language generation. Coreference resolution and its interconnection to other text understanding components.


Question topics for the State Doctoral Exams:
 
  1. Distributional word semantics, Word2Vec, Glove, and FastText models
  2. Language modelling
  3. Machine translation
  4. Seq2seq models and attention mechanism
  5. Question answering
  6. Convolutional neural networks for sentence classification
  7. Modeling contexts of use: Contextual representations and pretraining
  8. Transformers and self-attention for generative models 
  9. Natural language generation 
  10. Coreference resolution

Language of instruction

Czech

Mode of study

Not applicable.

Entry knowledge

Not applicable.

Rules for evaluation and completion of the course

Discussions within the lectures or individual consultations, a check of the prepared report.
Lectures and a preparation of a report.

Aims

To understand natural language processing and to learn how to apply basic algorithms in this field. To get acquainted with the algorithmic description of the main language levels: morphology, syntax, semantics, and pragmatics, as well as the resources of natural language data - corpora. To conceive basics of knowledge representation, inference, and relations to the artificial intelligence.
The students will get acquainted with natural language processing and will understand a range of neural network models that are commonly applied in the field. They will also grasp basics of neural implementations of attention mechanisms and sequence embedding models and how these modular components can be combined to build state of the art NLP systems. They will be able to implement and to evaluate common neural network models for various NLP applications.
Students will improve their programming skills and their knowledge and practical experience with tools for deep learning as well as with general processing of textual data.

Study aids

Not applicable.

Prerequisites and corequisites

Not applicable.

Basic literature

Not applicable.

Recommended reading

Géron, Aurélien. Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems. " O'Reilly Media, Inc.", 2017.
Goldberg, Yoav. "Neural network methods for natural language processing." Synthesis Lectures on Human Language Technologies 10, no. 1 (2017): 1-309.

Classification of course in study plans

  • Programme DIT Doctoral 0 year of study, winter semester, compulsory-optional
  • Programme DIT Doctoral 0 year of study, winter semester, compulsory-optional
  • Programme DIT-EN Doctoral 0 year of study, winter semester, compulsory-optional
  • Programme DIT-EN Doctoral 0 year of study, winter semester, compulsory-optional

  • Programme CSE-PHD-4 Doctoral

    branch DVI4 , 0 year of study, winter semester, elective

  • Programme CSE-PHD-4 Doctoral

    branch DVI4 , 0 year of study, winter semester, elective

  • Programme CSE-PHD-4 Doctoral

    branch DVI4 , 0 year of study, winter semester, elective

Type of course unit

 

Lecture

39 hod., optionally

Teacher / Lecturer

Syllabus

  1. Introduction, history of NLP, and modern approaches based on deep learning
  2. Word senses and word vector
  3. Dependency parsing
  4. Language models
  5. Machine translation
  6. Seq2seq models and attention
  7. Question answering
  8. Convolutional neural networks for sentence classification
  9. Information from parts of words: Subword models
  10. Modeling contexts of use: Contextual representations and pretraining
  11. Transformers and self-attention for generative models 
  12. Natural language generation 
  13. Coreference resolution

Guided consultation in combined form of studies

26 hod., optionally

Teacher / Lecturer