Move the emoji parsing to pageparser
authorBjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
Mon, 17 Dec 2018 20:03:23 +0000 (21:03 +0100)
committerBjørn Erik Pedersen <bjorn.erik.pedersen@gmail.com>
Thu, 20 Dec 2018 19:08:01 +0000 (20:08 +0100)
commit9cd54cab20a03475e34ca462bd943069111481ae
tree826eee0dbe8e7831ead2ace3b788286d1122fddd
parenta8853f1c5ace30ae8d256ad374bdb280c95d4228
Move the emoji parsing to pageparser

This avoids double parsing the page content when `enableEmoji=true`.

This commit also adds some general improvements to the parser, making it in general much faster:

```bash
benchmark                     old ns/op     new ns/op     delta
BenchmarkShortcodeLexer-4     90258         101730        +12.71%
BenchmarkParse-4              148940        15037         -89.90%

benchmark                     old allocs     new allocs     delta
BenchmarkShortcodeLexer-4     456            700            +53.51%
BenchmarkParse-4              28             33             +17.86%

benchmark                     old bytes     new bytes     delta
BenchmarkShortcodeLexer-4     69875         81014         +15.94%
BenchmarkParse-4              8128          8304          +2.17%
```

Running some site benchmarks with Emoji support turned on:

```bash
benchmark                                                                                     old ns/op     new ns/op     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     924556797     818115620     -11.51%

benchmark                                                                                     old allocs     new allocs     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     4112613        4133787        +0.51%

benchmark                                                                                     old bytes     new bytes     delta
BenchmarkSiteBuilding/TOML,num_langs=3,num_pages=5000,tags_per_page=5,shortcodes,render-4     426982864     424363832     -0.61%
```

Fixes #5534
13 files changed:
commands/convert.go
helpers/emoji.go
hugolib/page_content.go
hugolib/page_test.go
hugolib/pagebundler_handlers.go
hugolib/shortcode.go
parser/pageparser/item.go
parser/pageparser/pagelexer.go
parser/pageparser/pageparser.go
parser/pageparser/pageparser_intro_test.go
parser/pageparser/pageparser_main_test.go [new file with mode: 0644]
parser/pageparser/pageparser_shortcode_test.go
parser/pageparser/pageparser_test.go