On investigation, it appears that it's actually giflib that's linking to X11, and the obvious solution if it's a priority to build without X11, is to build without giflib. If giflib is needed, then its dependencies are needed too and that's not something FontAnvil can control. However, that raises a FontAnvil build-system bug: running configure --without-giflib on a system where giflib is actually installed, seems to build a binary that really does use giflib after all. After running configure --without-giflib and editing the Makefile to remove giflib from the LIBS variable, I confirmed that the resulting FontAnvil did not link to X11.
It appears to me that the build will work properly on a system that really does not have giflib nor X11; it's just using them because they are there. It's also possible that a binary built with giflib might work on a system that does not have X11; just mentioning the shared library in the binary's linkage does not necessarily mean it will really use that library at runtime. The non-functional --without-giflib option is a bug in the FontAnvil build, though.
mskala 
Tsukurimashou Font Family and IDSgrep
Over on the FontForge Github tracker, a user complains that FontAnvil is linking to X11 and shouldn't: https://github.com/fontforge/fontforge/issues/2580#issuecomment-159393874
On investigation, it appears that it's actually giflib that's linking to X11, and the obvious solution if it's a priority to build without X11, is to build without giflib. If giflib is needed, then its dependencies are needed too and that's not something FontAnvil can control. However, that raises a FontAnvil build-system bug: running configure --without-giflib on a system where giflib is actually installed, seems to build a binary that really does use giflib after all. After running configure --without-giflib and editing the Makefile to remove giflib from the LIBS variable, I confirmed that the resulting FontAnvil did not link to X11.
It appears to me that the build will work properly on a system that really does not have giflib nor X11; it's just using them because they are there. It's also possible that a binary built with giflib might work on a system that does not have X11; just mentioning the shared library in the binary's linkage does not necessarily mean it will really use that library at runtime. The non-functional --without-giflib option is a bug in the FontAnvil build, though.