Treesitter lockfile read causes nvim flicker on startup

[michaelPotter]

Describe the bug

I've noticed a "flicker" when nvim is starting up. This only occurs when I have orgmode installed and a healthy number of other plugins installed. Without orgmode installed the flicker goes away; the fewer other plugins I have installed, the more the flicker is reduced until it is unnoticeable or nonexistant.

I've done some digging and I think it is due to an interaction between contention on the main thread from other plugins, the way lazy.nvim loads VeryLazy plugins, and orgmode's use of vim.wait to block the plugin's setup() method until the async lockfile read is finished.

In other words: I think the flicker is a lazy.nvim artifact caused by slow-loading plugins vim.wait(), and orgmode's reading of the treesitter lockfile is being slowed down by contention on the main thread.

checkhealth

==============================================================================
orgmode: ✅

Orgmode ~

• ✅ OK Treesitter grammar installed (version 2.0.0)
• ✅ OK Setup called
• ✅ OK org_agenda_files configured
• ✅ OK org_default_notes_file configured

Steps to reproduce

Using the provided init.lua...

1. Start nvim to install the plugins and stuff nvim -u minimal_init.lua
2. Exit nvim
3. Run nvim again. You'll probably see a flicker.
   a. Might try restarting a few times; sometimes the flicker is more visible.
   b. There's a number in the config that can increase contention to make the flicker more visible
4. Comment out the orgmode plugin spec from minimal_init.lua
5. Run nvim again. You shouldn't see the flicker

Expected behavior

No flicker

Emacs functionality

No response

Minimal init.lua

minimal_init.lua

local tmp_dir = vim.env.TMPDIR or vim.env.TMP or vim.env.TEMP or '/tmp'
local nvim_root = tmp_dir .. '/nvim_orgmode'
local lazy_root = nvim_root .. '/lazy'
local lazypath = lazy_root .. '/lazy.nvim'

for _, name in ipairs({ 'config', 'data', 'state', 'cache' }) do
  vim.env[('XDG_%s_HOME'):format(name:upper())] = nvim_root .. '/' .. name
end

-- Install lazy.nvim if not already installed
if not vim.uv.fs_stat(lazypath) then
  vim.fn.system({
    'git',
    'clone',
    '--filter=blob:none',
    'https://github.com/folke/lazy.nvim.git',
    '--branch=stable', -- latest stable release
    lazypath,
  })
end
vim.opt.rtp:prepend(lazypath)


local function do_work_async()
  -- Force some work to occur on the main thread; since this is scheduled, it
  -- doesn't block lazy from completing this plugin's load step, but will slow
  -- down orgmode's plugin load.
  vim.schedule(function()
    local i = 0
    -- Tweak this number to increase/decrease flicker
    local max = 1000 * 1000 * 50
    for i=1, max do
      i = i + 1
    end
  end)
end


require('lazy').setup({
  {
    'nvim-orgmode/orgmode',
    event = 'VeryLazy',
    ft = { 'org' },
    config = function()
      -- -- WORKAROUND: wrapping in vim.schedule also seems to make the flicker go away. Unclear if there are side effects such as failure to install treesitter grammar
      -- vim.schedule(function()
        require('orgmode').setup()
      -- end)
    end,
  },
  -- A series of tpope plugins whose configs do some work to simulate having a bunch more plugins
  { 'tpope/vim-abolish', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-afterimage', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-capslock', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-characterize', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-classpath', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-commentary', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-cucumber', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-dadbod', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-dispatch', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-endwise', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-fireplace', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-flagship', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-flatfoot', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-fugitive', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-git', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-haml', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-haystack', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-heroku', event='VeryLazy', config=do_work_async },
  { 'tpope/vim-jdaddy', event='VeryLazy', config=do_work_async },
}, {
  root = lazy_root,
  lockfile = nvim_root .. '/lazy.json',
  dev = { path = "~/src/nvim_plugins" },
  -- install = {
  --   missing = false,
  -- },
})

-- require('lazy').sync({
--   wait = true,
--   show = false,
-- })

require('lazy').profile()

Here is an example Lazy.nvim profile output using the repro.lua where the flicker/orgmode time was particularly slow:

Home (H) Install (I) Update (U) Sync (S) Clean (X) Check (C) Log (L)
Restore (R) Profile (P) Debug (D) Help (?)

Startuptime: 16.74ms

Based on the actual CPU time of the Neovim process till UIEnter.
This is more accurate than nvim --startuptime.
LazyStart 9.39ms
LazyDone 12.94ms (+3.55ms)
UIEnter 16.74ms (+3.81ms)

Profile

You can press to change sorting between chronological order & time taken.
Press to filter profiling entries that took more time than a given threshold

●  lazy.nvim 2.64ms
  ➜  module 1.56ms
  ➜  config 0.21ms
  ➜  spec 0.33ms
    ★  pkg 0.02ms
    ★  resolve plugins 0.09ms
  ➜  state 0.05ms
  ➜  install 0.01ms
  ➜  handlers 0.15ms
●  startup 1.46ms
  ➜  runtime/filetype.lua 0.11ms
  ➜  init 0ms
  ➜  start 0ms
  ➜  rtp plugins 1.32ms
    ★  runtime/plugin/editorconfig.lua 0.07ms
    ★  runtime/plugin/gzip.vim 0.12ms
    ★  runtime/plugin/man.lua 0.04ms
    ★  runtime/plugin/matchit.vim 0.22ms
    ★  runtime/plugin/matchparen.vim 0.08ms
    ★  runtime/plugin/netrwPlugin.vim 0.23ms
    ★  runtime/plugin/osc52.lua 0.05ms
    ★  runtime/plugin/rplugin.vim 0.12ms
    ★  runtime/plugin/shada.vim 0.04ms
    ★  runtime/plugin/spellfile.vim 0.02ms
    ★  runtime/plugin/tarPlugin.vim 0.06ms
    ★  runtime/plugin/tohtml.lua 0.06ms
    ★  runtime/plugin/tutor.vim 0.02ms
    ★  runtime/plugin/zipPlugin.vim 0.08ms
  ➜  after 0.01ms
●  startuptime 16.74ms
●  VeryLazy 218.65ms
  ➜  vim-characterize 0.14ms
    ★  vim-characterize/plugin/characterize.vim 0.07ms
  ➜  vim-classpath 0.11ms
    ★  vim-classpath/plugin/classpath.vim 0.04ms
  ➜  vim-commentary 0.13ms
    ★  vim-commentary/plugin/commentary.vim 0.07ms
  ➜  vim-cucumber 0.11ms
    ★  vim-cucumber/ftdetect/cucumber.vim 0.02ms
  ➜  vim-dadbod 0.16ms
    ★  vim-dadbod/plugin/dadbod.vim 0.07ms
  ➜  vim-dispatch 0.78ms
    ★  vim-dispatch/plugin/dispatch.vim 0.67ms
  ➜  vim-endwise 0.24ms
    ★  vim-endwise/plugin/endwise.vim 0.17ms
  ➜  vim-fireplace 0.12ms
    ★  vim-fireplace/plugin/fireplace.vim 0.05ms
  ➜  vim-abolish 0.52ms
    ★  vim-abolish/plugin/abolish.vim 0.42ms
  ➜  vim-flatfoot 0.23ms
    ★  vim-flatfoot/plugin/flatfoot.vim 0.11ms
  ➜  vim-fugitive 0.64ms
    ★  vim-fugitive/plugin/fugitive.vim 0.52ms
    ★  vim-fugitive/ftdetect/fugitive.vim 0.04ms
  ➜  vim-git 0.21ms
    ★  vim-git/ftdetect/git.vim 0.09ms
  ➜  vim-haml 0.15ms
    ★  vim-haml/ftdetect/haml.vim 0.03ms
  ➜  vim-flagship 0.12ms
    ★  vim-flagship/plugin/flagship.vim 0.04ms
  ➜  vim-haystack 0.12ms
    ★  vim-haystack/plugin/haystack.vim 0.03ms
  ➜  orgmode 213.85ms
  ➜  vim-afterimage 0.24ms
    ★  vim-afterimage/plugin/afterimage.vim 0.14ms
  ➜  vim-capslock 0.25ms
    ★  vim-capslock/plugin/capslock.vim 0.12ms
  ➜  vim-jdaddy 0.14ms
    ★  vim-jdaddy/plugin/jdaddy.vim 0.04ms
  ➜  vim-heroku 0.21ms
    ★  vim-heroku/plugin/heroku.vim 0.12ms
  ➜  dadbod 0ms
  ➜  fireplace 0ms
  ➜  fugitive 0ms
  ➜  capslock 0.01ms

Screenshots and recordings

No response

nvim-orgmode version

468570f

OS / Distro

linux endeavour os

Neovim version/commit

v0.11.2

Additional context

I was able to sort of work around this by wrapping orgmode.setup() with vim.schedule... unclear if there are side effects to this:
This workaround doesn't work if the first opened file is an org file.

      vim.schedule(function()
        require('orgmode').setup()
      end)

Also, it looks like the actual lockfile read happens asynchronously with the builtin Promise library, but the returned promise is blocked with :wait() which blocks the setup() method. Seems like making the entire treesitter check/install process async which might help with this?

[michaelPotter]

I've done some more digging and tried some approaches.

Making the TS install completely async won't work since the ftplugin will fail before the parser is installed if you open an org file at the same time the plugin is installed.

I believe the issue is related to the use of vim.wait(); I was able to repro the flickering with 1 dummy plugin using vim.wait() in a similar manner, and several other plugins providing contention.

I was able to resolve the issue by setting vim.wait()'s fourth parameter "fast_only" to true, down in orgmode.utils.promise's Promise.wait():

  local success, code = vim.wait(timeout, function()
    return is_done
  end, 1, true)

Unfortunately this causes other things to break. Maybe there could be a way to set it optionally? Food for thought.

[kristijanhusak]

Thanks for investigating. I didn't notice this since I lazy load on demand, but I did notice the flickering. I pushed an update to use io to read and write lock file. This should make flicker go away. Please give it a test and let me know if it works for you.

[michaelPotter]

Works great! Thanks much!