Release Notes

This release is meant to fix the following issues:

• Fix incorrect masking of ConstituencyCRF
• Handle single-root case in MatrixTree
• Retain the computational graph for marginals

We also add implementations of sparsemax and SparsemaxSemiring (see details in Martins et al. 2016 and Rush et al. 2020).

Release Notes

Highlights

We add implementations of many structured prediction algorithms presented in the form of torch.disrtibutions and semiring notations[^1], including

• Tree
  • MatrixTree (Koo et al., 2007; Ma and Hovy, 2017)
  • DependencyCRF (Eisner et al., 2000; Zhang et al., 2020)
  • Dependency2oCRF (McDonald et al., 2006; Zhang et al., 2020)
  • ConstituencyCRF (Stern et al. 2017; Zhang et al., 2020b)
• Linear Chain:
  • LinearChainCRF (Lafferty et al., 2001)

Take LinearChainCRF as an example:

>>> from supar import LinearChainCRF
>>> batch_size, seq_len, n_tags = 2, 5, 4
>>> lens = torch.tensor([3, 4])
>>> value = torch.randint(n_tags, (batch_size, seq_len))
>>> s1 = LinearChainCRF(torch.randn(batch_size, seq_len, n_tags),
                        torch.randn(n_tags+1, n_tags+1),
                        lens)
>>> s2 = LinearChainCRF(torch.randn(batch_size, seq_len, n_tags),
                        torch.randn(n_tags+1, n_tags+1),
                        lens)
>>> s1.max
tensor([4.4120, 8.9672], grad_fn=<MaxBackward0>)
>>> s1.argmax
tensor([[2, 0, 3, 0, 0],
        [3, 3, 3, 2, 0]])
>>> s1.log_partition
tensor([ 6.3486, 10.9106], grad_fn=<LogsumexpBackward>)
>>> s1.log_prob(value)
tensor([ -8.1515, -10.5572], grad_fn=<SubBackward0>)
>>> s1.entropy
tensor([3.4150, 3.6549], grad_fn=<SelectBackward>)
>>> s1.kl(s2)
tensor([4.0333, 4.3807], grad_fn=<SelectBackward>)

Bug fixes

• Fix bug of model saving (#82)
• Fix issue of stride setting for small BERT (#86)
• Fix preprocessing crashes for some UD treebanks (#85)

[^1]: The implementations of structured distributions and semirings are heavily borrowed from torchstruct with some tailoring. For more details, see their tutorial paper and Goodman's paper.

Release Notes

• ELMo support
• Checkpoint support
• Dataloader now yields Batch objects
• Fix con name conflict in Windows system (#74)

Release Notes

New Features

• Built-in tokenizers (#47)
• Variational Inference methods
  • Mean Field Variational Inference
  • Loopy Belief Propagation
• Semantic Dependency Parsers
  • Biaffine
  • MFVI/LBP
• Model finetuning

Bug Fixes

• Fix issue of building nltk.Tree from string with parentheses (#59, #65)

Available Parsers

• Dependency Parser
  • Biaffine (Dozat and Manning, 2017)
  • CRF/MatrixTree (Koo et al., 2007; Ma and Hovy, 2017)
  • CRF2o (Zhang et al., 2020a)
• Constituency Parser
  • CRF (Zhang et al., 2020b)
• Semantic Dependency Parser
  • Biaffine (Dozat and Manning, 2018)
  • MFVI/LBP (Wang et al, 2019)

Release Notes

Released Models

The following parsers are released in SuPar package and the corresponding English/Chinese pretrained models can be found in the attachments.

• Dependency Parser
  • Biaffine Dependency Parser (Dozat and Manning, 2017)
  • Non-Projective CRF Dependency Parser with Matrix Tree (Koo et al., 2007; Ma and Hovy, 2017)
  • Projective First-Order CRF Dependency Parser (Zhang et al., 2020a)
  • Projective Second-Order Dependency Parser (Zhang et al, 2020a)
• Constituency Parser
  • CRF Constituency Parser (Zhang et al, 2020b)

References

• Timothy Dozat and Christopher D. Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing.
• Terry Koo, Amir Globerson, Xavier Carreras and Michael Collins. 2007. Structured Prediction Models via the Matrix-Tree Theorem.
• Xuezhe Ma and Eduard Hovy. 2017. Neural Probabilistic Model for Non-projective MST Parsing.
• Yu Zhang, Houquan Zhou and Zhenghua Li. 2020. Fast and Accurate Neural CRF Constituency Parsing.
• Yu Zhang, Zhenghua Li and Min Zhang. 2020. Efficient Second-Order TreeCRF for Neural Dependency Parsing.