Week 7, Dialogue Systems

Content

• Review of homeworks with Amazon Alexa:
  • Let’s check the Metal Archives Lookup
  • Preview others via Amazon Developer Console
• Feedback on ParlAI
  • What’s good?
  • What’s unclear?
  • Would you like to use it for research?
• Deciding which research papers to follow

Personal notes to two of the suggested papers

Unsupervised NLU – clustering dialogue intents and slots without labeled data by Shi et al.

• Idea can be separated into:
  • How the clustering is done?
    • Mapping using autoencoder to fixed size embedding
    • Features participating in the final embedding
      • BiTerm model for intent clustering Yan et al. 2013 (interesting – should be better than LDA & PLSA)
      • Word embeddings
      • Frequent word feature (interesting) - clustered noun words
    • RBF function used as distance measure for two clusters
    • Intent clustering uses the Biterm and the word word features, Slot clustering uses only the word features
  • How to choose optimal number of clusters?
    • using a dataset split for choosing optimal number of clusters (stopping criterion)
• Very much needed approach
• Weak baselines, experiments results are promising but probably far from usable
  • Experiment results can be improved in supervised way
  • Manual mapping between clusters and named intents and slot values works surprisingly well for evaluation. Is it a coincidence for this particular dataset?

End-to-end task-oriented dialogue systems – seq2seq with attention for knowledge base queries by Wen et al.

• Works surprisingly well on Wizard of Oz KWRET database
• Solves end-to-end training of task-oriented dialogue with a Knowledge Base in form of table
• Architecture especially interesting for soft attention on KB
  • dialogues history encoded LSTM word by word (both user & agent utterances)
  • slots are defined by column in database
  • pick from database is represented as probability distribution if the KB entry will be used
    • unclear how exactly it works
    • Why it works so well> – I worked on very similar architecture but gradient was not propagated
  • Creates matrix U with dimension of number of (slots/KB columns x number of turns)
  • Decoder uses attention over input and attention over KB through matrix U
• Training uses XE loss and REINFORCE algorithm for fine-tuning
  • REINFORCE trained in second stage
  • XE loss helps REINFORCE a lot for bootstrapping (sampling from with a strong prior)
• Would be interesting to compare with Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning: J. Williams et al

Relevant datasets

• MultiWOZ – multi-domain dataset for city information (was used in the state-tracking paper mentioned above)
• KWRET – in-car dialogues about weather, scheduling, and navigation

Homework

1. (3 points) :
   • Write a project plan draft for implementing your chosen research paper.
     • What is a project plan? It is as detailed plan as possible how to achieve the project’s goal. Typically it is realized by decomposing the goal into smaller goals/tasks and estimating their effort. Note that some tasks are dependent on each other, some can be parallelized. Some are critical, some are optional.
     • Please read How to effectively scope your software projects.
     • Gitlab contains some tools which helps with project planning. See project management feature describtion to get known the vocabulary if interested.
     • Submit the homework as Merge Request to the dias-e2e repository. Extend the plan.md and rename the file so it contains your name e.g. plan-ondra.md.

2. (2 BONUS points) Choose any research article referenced in this lab and make a summary. Summary should include simplified explanation of the core idea, discuss validity of experiments, how relevant is the article and future directions. Be prepared to discuss the summary with your fellow colleagues. Feel free to submit such summaries of papers anytime during rest of the semester.

3. (1 BONUS point) for implementing one or more task from your plan. (Multiple points may be awarded if you implement more than one non-trivial task until next lab)