My 2023 Emacs Python Setup

Goodbye Elpy, Goodbye Projectile

Elpy has been the primary mode for Python development for me for years now. But sadly, it looks like the project is no more. The good news is there are better ways to do what it did. It’s bittersweet to say goodbye to it and I will be eternally grateful to the authors, but progress is progress.

Similarly, Projectile was what I used to manage projects. But now Emacs has project.el built in and I’ve opted to use that instead. One nice thing about project.el is it uses other standard stuff underneath like xref. I configured xref to use Ripgrep and now the Project commands like C-x p g use Ripgrep:

1(use-package xref
2  :config
3  (setq xref-search-program 'ripgrep))

Native builds and tree-sitter

I always build Emacs myself from source if I can. I run Gentoo on my personal computer so that goes without saying, but I do it on Ubuntu too, if only to get the latest versions. This does mean I can easily enable a couple of new features: native builds and tree-sitter.

On Gentoo this was a simple as adding a couple of USE flags to portage. My USE flags for emacs now look like:

app-editors/emacs athena cairo gui gtk harfbuzz json libxml2 source tree-sitter jit -X

The gtk -X also implies a pgtk build which is nice because I use wayland (sway).

On Ubuntu (20.04, yeah, old, this is one reason I prefer rolling distros) it was more difficult. I first pulled the source code (emacs-29.1.tar.gz) from a nearby GNU mirror per the GNU website. Then a few packages are required (I use i3/X11 on Ubuntu):

1sudo apt install autoconf make gcc texinfo libgtk-3-dev libxpm-dev libjpeg-dev \
2     libgif-dev libtiff5-dev libgnutls28-dev libncurses5-dev libjansson-dev \
3     libharfbuzz-dev libharfbuzz-bin imagemagick libmagickwand-dev libgccjit-10-dev \
4     libgccjit0 gcc-10 libjansson4 libjansson-dev xaw3dg-dev texinfo libx11-dev

Now, because libgccjit (required for native builds) is only for GCC 10, the build process has to be configured for GCC 10 specifically, in addition to enabling all the wanted features:

1CC="gcc-10" ./configure --prefix=$HOME --with-json --with-native-compilation=aot \
2  --with-modules --with-compress-install --with-threads --with-included-regex \
3  --with-x-toolkit=lucid --with-zlib --with-jpeg --with-png --with-imagemagick \
4  --with-tiff --with-xpm --with-gnutls --with-xft --with-xml2 --with-mailutils \
5  --with-tree-sitter

Note that I keep my own builds in $HOME by setting --prefix. By default the installation would put it in the system directories which I prefer not to do as those are controlled by my system package manager. Also note that I set --with-native-compilation=aot which makes native builds ahead of time instead of JIT compiling them. Run ./configure --help to see all of the build options.

Then I just compiled it:

1make -j 8                       # 8 threads

The build can be tested with src/emacs -Q then, if it works:

1make install

Eglot

Elpy provided a proper IDE experience for Python but it did it in a completely custom, albeit very clever, way via a special RPC process which used jedi. Now with LSP we can get essentially the same sort of thing but in a more standard way that works with all languages.

I have tried LSP (in particular, lsp-mode) in emacs before, but I wasn’t impressed. I cannot stand latency and the moment I detect latency when merely typing in a text editor, I walk away. But I’m pleased to say that with Emacs 29, native builds, Eglot and python-lsp-server it is now fast enough for me. lsp-mode might very well be fast enough now too. I’ll probably try it eventually.

I installed python-lsp-server (with pipx on Ubuntu). This is my preferred way of installing Python apps if they’re not available in the base distro. Notice how there will be only one LSP server installed for my whole system (not one per virtualenv).

Enabling Eglot is easy. To make it work for Python it just needs the following:

1(use-package eglot
2  :hook (python-mode . eglot-ensure))

Now just open a Python file and it should work. It does everything Elpy did (or, at least, what I used it for) and more. Just like that.

By default, Eglot uses Flymake. I had previously been using Flycheck. I can’t really remember why, to be honest, so I’ll try using Flymake instead and say goodbye to Flycheck for now too.

Virtualenvs

OK, so, it doesn’t completely just work. One of the most important things for me is being able to jump to the definition of a symbol in the source. This does just work for first party stuff and (kinda) for standard library stuff, but it won’t work for third party stuff. That’s because the LSP server doesn’t know where to find those libraries.

Usually when developing on a Python project one would create a virtual environment for it. I make everything a package such that doing a pip install -e . installs the package and all of its dependencies into the virtualenv. Then you just need to make the LSP server aware of this environment.

I used to use virtualenvwrapper to create virtualenvs for each project, but I’ve found a better way: direnv. This allows you to create .envrc files in directories with anything you want in it then automatically loads it into your environment when you change to that directory. What’s even neater is it has built-in support for Python (and other languages).

To install direnv on Gentoo I used the Guru overlay:

1eselect repository enable guru

After installing and setting up direnv, make a file called .envrc at the top of your project and put the following:

1layout python

That’s it! After enabling your project for direnv support it will automatically create a virtualenv and activate it. When you change directory, it will deactivate it. Amazing!

In Emacs you can install the direnv package and enable it:

1(use-package direnv
2  :config
3  (direnv-mode))

Now when you browse to a project with a .envrc file it will just work.

Tree-sitter

Finally, to enable tree-sitter I needed to first install the grammar for Python, I added the following to my emacs config:

1(setq treesit-language-source-alist
2   '((python "https://github.com/tree-sitter/tree-sitter-python")))

And then (after evaling the above) you can run: M-x treesit-install-language-grammar. This builds the grammar for you and puts it in your emacs config.

Now you can use the mode python-ts-mode instead of python-mode:

1(use-package python
2  :mode ("\\.py\\'" . python-ts-mode)
3  :init
4  (add-to-list 'major-mode-remap-alist '(python-mode . python-ts-mode)))

The major-mode-remap-list entry means python-ts-mode will be used whenever python-mode would have been used, like when opening a script with no file extension but a Python shebang.

Completion

One thing I cannot do without is some kind of completion capability. In bash I use tab-completion extensively and I consider any keyboard-driven software that doesn’t support at least tab-completion to be defective.

Basic completion is supported in Emacs out of the box but it can be extended to be quite sophisticated. But I’ve always found it a bit overwhelming. My life was changed when I first enabled ido. The combination of completion and narrowing is amazing. Later I switched to other packages like ivy, Helm and Selectrum and enabled in-buffer completion with Company. Selectrum is now defunct and replaced with Vertico.

For the first time, I have a completion set up that I understand and that I’m very happy with.

What I really wanted was fuzzy-style completion in minibuffer contexts but dead basic prefix-style completion within buffers. I also want the completion within-buffer to be driven by the tab key like in a bash shell. I’ve settled on Company within-buffer and Vertico in the minibuffer.

I like the setting (setq tab-always-indent 'complete) which causes TAB to indent first, then complete, but I was getting weird behaviour where that completion would not launch company. So instead:

1(global-set-key (kbd "TAB") #'company-indent-or-complete-common)

This now does the right thing but launches Company instead of the default completion function.

The other major part is completion styles. I like the Orderless style for the fuzzy minibuffer style, but it doesn’t work for basic completion. Emacs supports setting a list of completion styles by setting completion-styles and further refining that for specific categories by setting completion-category-overrides. The trouble is, the category names for the latter are quite hard to find. But eventually I settled on the following configuration:

1(use-package orderless
2  :init
3  (setq completion-styles '(basic partial-completion orderless)
4        completion-category-defaults nil
5        completion-category-overrides '((project-file (styles orderless))
6                                        (buffer (styles orderless))
7                                        (command (styles orderless)))))

This sets basic and partial-completion styles first by default everywhere. Company doesn’t really support the Orderless style, which is fine by me as I don’t want it in-buffer anyway. I then override it for particular categories to add orderless to the front. project-file is for finding files in projects with C-x p f, buffer is for switching buffers and command is for running commands with M-x.

Conclusion

To sum up, I’ve switched from Projectile to project.el, from Elpy to Eglot/LSP and from virtualenvwrapper to direnv as well as including the latest improvements like native builds and tree-sitter. This has really simplified my config and I seem to have a renewed love for Emacs.

I’ve been using this configuration for a few days now for real work and I really love it so far. Things like the eldoc and xref jump to definition features are working perfectly now and I’ve had real trouble with consistent behaviour before.

My actual emacs config does include a number of extra tweaks to all of this stuff. I love reading other people’s .emacs files, so maybe you’ll enjoy reading mine too: https://github.com/georgek/dot-emacs

Happy hacking!