To install this package, run in Emacs:
M-x package-install RET kiwix RET
* Intro
Searching offline Wikipedia through Kiwix.
[[kiwix.el Ivy async completion.png]]
[[kiwix.el with EWW.png]]
This =kiwix.el= supports query =kiwix-tools='s =kiwix-serve= server through URL API.
The =kiwix-serve= server can be started from command-line if you have =kiwix-tools=
installed, or from Docker container [fn:1].
* Install
** Install Kiwix
*** Docker
Reference this issue as background info: https://github.com/kiwix/kiwix-tools/issues/257
#+begin_src sh :eval no
docker pull kiwix/kiwix-serve
#+end_src
*** Flatpak
:PROPERTIES:
:URL: https://wiki.kiwix.org/wiki/Flatpak
:END:
#+begin_src org
,#+begin_src sh :dir /sudo::/tmp
# Install Flatpak (on Debian/Ubuntu)
sudo pacman -S flatpak
# Install Flathub (for the dependencies)
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
# Download the Kiwix Desktop Flatpak
wget https://download.kiwix.org/release/kiwix-desktop/org.kiwix.desktop.2.0-beta2.flatpak
,#+end_src
,#+begin_src sh :dir /tmp :eval no
# Install Kiwix Desktop
flatpak install org.kiwix.desktop.2.0-beta2.flatpak
,#+end_src
,#+begin_src sh :eval no
# Run Kiwix Desktop (but Kiwix should be available through your app launcher anyway)
flatpak run org.kiwix.desktop
,#+end_src
#+end_src
*** Download
- https://sourceforge.net/projects/kiwix
- http://www.kiwix.org/wiki/Software
*** Linux
**** Arch
#+begin_src org
,#+begin_src sh :dir /sudo:: :results none
aurman -S --noconfirm kiwix-bin
,#+end_src
#+end_src
*** Web Browser
**** Firefox
https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/
**** Chrome
https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk
** MELPA
#+begin_src emacs-lisp :eval no
(use-package kiwix
:ensure t
:after org
:commands (kiwix-launch-server kiwix-at-point)
:custom ((kiwix-server-use-docker t)
(kiwix-server-port 8089)
(kiwix-default-library "wikipedia_en_all_2016-02.zim"))
:hook (org-load . org-kiwix-setup-link))
#+end_src
* Setup
If you use kiwix-serve Docker container, you can create an Systemd unit service
to auto start Docker container. Here is the systemd unit config file:
** Dockerize kiwix-tools (kiwix-serve, etc)
:PROPERTIES:
:URL: https://hub.docker.com/r/kiwix/kiwix-serve
:ISSUE: https://github.com/kiwix/kiwix-tools/issues/257
:Pull-Request: https://github.com/kiwix/kiwix-tools/pull/268
:Attachments: screenshot_1.png screenshot_2.png
:ID: e82e194f-2cc8-45eb-a378-f8bd6d7c6b1a
:END:
#+begin_src sh :async
docker pull kiwix/kiwix-serve
#+end_src
#+RESULTS[<2019-03-24 08:33:29> ace542940af6e465f90f0a3a8515e876fd267ad5]:
#+begin_example
Using default tag: latest
latest: Pulling from kiwix/kiwix-serve
8e402f1a9c57: Pulling fs layer
7024865ce0e2: Pulling fs layer
ad4c9cfc45dc: Pulling fs layer
c4d62acdb073: Pulling fs layer
c4d62acdb073: Waiting
8e402f1a9c57: Verifying Checksum
8e402f1a9c57: Download complete
7024865ce0e2: Verifying Checksum
7024865ce0e2: Download complete
8e402f1a9c57: Pull complete
7024865ce0e2: Pull complete
c4d62acdb073: Verifying Checksum
c4d62acdb073: Download complete
ad4c9cfc45dc: Verifying Checksum
ad4c9cfc45dc: Download complete
ad4c9cfc45dc: Pull complete
c4d62acdb073: Pull complete
Digest: sha256:8837effa1a4fce750dc373d58b47063b368228331ccacb267c6ae7d3e311e66c
Status: Downloaded newer image for kiwix/kiwix-serve:latest
#+end_example
https://github.com/kiwix/kiwix-tools/blob/master/docker/server/Dockerfile
#+begin_src org
,#+begin_src dockerfile
FROM alpine:latest
LABEL maintainer Emmanuel Engelhart <kelson@kiwix.org>
# Install kiwix-serve
WORKDIR /
RUN apk add --no-cache curl bzip2
RUN curl -kL https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-x86_64-1.1.0.tar.gz | tar -xz && \
mv kiwix-tools*/kiwix-serve /usr/local/bin && \
rm -r kiwix-tools*
# Configure kiwix-serve
VOLUME /data
ENV PORT 80
EXPOSE $PORT
# Run kiwix-serve
WORKDIR /data
ENTRYPOINT ["/usr/local/bin/kiwix-serve", "--port", "$PORT"]
,#+end_src
#+end_src
How to run?
Given =wikipedia.zim= ([[#ZIM][Zim database files]]) resides in =/tmp/zim/=, execute the
following command:
#+begin_src sh :eval no
# if you don't have libraries index file "library.xml"
docker container run -d --name kiwix-serve -v /tmp/zim:/data -p 8080:80 kiwix/kiwix-serve wikipedia.zim
# if you have libraries index file "library.xml"
docker container run -d --name kiwix-serve -v /tmp/zim:/data -p 8080:80 kiwix/kiwix-serve --library library.xml
#+end_src
*NOTE*: You can generate the libraries index file "library.xml" with following command:
#+begin_src sh
cd ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library/
for zim in $(ls *.zim); do
kiwix-manage library.xml add $zim
done
#+end_src
*NOTE*: Using the libraries index file method, you can have all libraries served
in Docker container instead of just one library.
If you put ZIM files in other places not =/tmp/zim/=, you can use follow my command:
#+NAME: create kiwix-serve container with custom port
#+begin_src sh :session "*kiwix-serve*"
docker container run -d \
--name kiwix-serve \
-v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data \
-p 8089:80 \
kiwix/kiwix-serve wikipedia_zh_all_2015-11.zim
#+end_src
Visit http://localhost:8080 or http://localhost:8089 (if you exposed different
port).
For easy launch the docker run command, you can add command alias in shell profile:
#+begin_src shell :eval no
alias kiwix-docker-wikipedia_zh_all="docker container run --name kiwix-serve -d -v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data -p 8089:80 kiwix/kiwix-serve wikipedia_zh_all_2015-11.zim"
alias kiwix-docker-wikipedia="docker container run --name kiwix-serve -d -v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data -p 8089:80 kiwix/kiwix-serve wikipedia.zim"
#+end_src
*** create a systemd unit for kiwix-serve Docker service
#+begin_src org
,#+begin_src systemd :tangle "~/.config/systemd/user/kiwix-serve.timer"
[Unit]
Description=Start kiwx-serve Docker container server at system startup after 5 minutes
[Timer]
OnBootSec=2min
Unit=kiwix-serve.service
[Install]
WantedBy=default.target
,#+end_src
,#+begin_src systemd :tangle "~/.config/systemd/user/kiwix-serve.service"
[Unit]
Description=kiwix-serve Docker server
After=docker.service
[Service]
Type=simple
ExecStart=/usr/bin/docker container start -i kiwix-serve
ExecStop=/usr/bin/docker container stop kiwix-serve
[Install]
WantedBy=default.target
,#+end_src
#+end_src
*NOTE*: You need to use option =-i= for =docker container start= command to avoid
systemd auto exit and stop =kiwix-serve= container.
#+begin_src sh :results output
systemctl --user enable kiwix-serve.timer
systemctl --user status kiwix-serve.timer | cat
#+end_src
#+RESULTS[<2019-03-24 11:45:40> 6470584177f091e79067f9fd96a97c340e00a41f]:
: ● kiwix-serve.timer - Start kiwx-serve Docker container server at system startup after 5 minutes
: Loaded: loaded (/home/stardiviner/.config/systemd/user/kiwix-serve.timer; enabled; vendor preset: enabled)
: Active: inactive (dead)
: Trigger: n/a
#+begin_src sh
systemctl --user start kiwix-serve.service
#+end_src
#+begin_src sh
systemctl --user status kiwix-serve.service | cat
#+end_src
#+RESULTS[<2019-03-24 12:00:49> 10a33f8521fa2c72e8c1107559e1fb18b58d7da2]:
: ● kiwix-serve.service - kiwix-serve Docker server
: Loaded: loaded (/home/stardiviner/.config/systemd/user/kiwix-serve.service; disabled; vendor preset: enabled)
: Active: active (running) since Sun 2019-03-24 12:00:14 CST; 34s ago
: Main PID: 2587 (docker)
: CGroup: /user.slice/user-1000.slice/user@1000.service/kiwix-serve.service
: └─2587 /usr/bin/docker container start -i kiwix-serve
:
: Mar 24 12:00:14 dark systemd[694]: Started kiwix-serve Docker server.
*NOTE*: Because =kiwix-serve.service= use command =docker container start
kiwix-serve=, so that the container =kiwix-serve= must already been created by
[[create kiwix-serve container with custom port][this command]], you can check whether the container is created:
#+begin_src sh :results output
docker container ls | head -n 1
docker container ls --all | grep "kiwix-serve" | cat
#+end_src
#+RESULTS[<2019-03-24 11:50:36> e28015e8e78015623bd53ae596015949dc80c549]:
: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
: b47533ecd7f6 kiwix/kiwix-serve "/usr/local/bin/kiwi…" 3 hours ago Exited (137) 2 minutes ago kiwix-serve
: e2f201e655ac kiwix/kiwix-serve "/usr/local/bin/kiwi…" 3 hours ago Created distracted_hofstadter
* Load
** use-package
#+begin_src emacs-lisp
(use-package kiwix
:ensure t
:after org
:commands (kiwix-launch-server kiwix-at-point)
:init (require 'org-kiwix)
(setq kiwix-server-use-docker t
kiwix-server-port 8089
kiwix-default-library "wikipedia_en_all_2016-02.zim" ; "wikipedia_zh_all_2015-11.zim"
kiwix-default-browser-function 'eww))
#+end_src
* Usage
** Use in Emacs
=[M-x kiwix-at-point]=
** Org Mode integration
=[C-c C-l]= to insert link.
The link format is like this:
#+BEGIN_EXAMPLE
[[wikipedia:(library):search][description]]
#+END_EXAMPLE
The =(library)= can be =wikipedia_en=, =wikipedia_zh=, =wiktionary_en=, or =en=, =zh= etc.
** EWW integration
Set following option in your config to use EWW in Emacs as your default _for
Kiwix only_.
#+begin_src emacs-lisp
(setq kiwix-default-browser-function 'eww-browse-url)
#+end_src
#+RESULTS[<2019-10-15 18:32:09> 3b9749599d792fb0ea5cd3566095ae16f1fc7f30]:
: eww-browse-url
[[kiwix.el with EWW.png]]
** Async search completion keywords candidates
[[kiwix.el Ivy async completion.png]]
* Changelog
** DONE implemented async instantly input suggestion completion in Ivy
CLOSED: [2019-10-08 Tue 22:07]
:LOGBOOK:
- State "DONE" from [2019-10-08 Tue 22:07]
:END:
This feature is very subtle :)
* Test
- [[wikipedia:Operations%20Research][Operations Research]] :: query contains space.
- [[wikipedia:Operations%20research][Operations research]] :: the second word is not capitalized.
- [[wikipedia:%E4%B8%AD%E5%9B%BD][中国]] :: non-english query
- [[wikipedia:meta-circular%20interpreter][meta-circular interpreter]] :: only capitalize the first word.
* How does this extension work?
** integrate with Emacs
*** core
I found Kiwix will return a URL like this:
#+BEGIN_EXAMPLE
http://127.0.0.1:8000/wikinews_en_all_2015-11/A/Big_Linux_Beta_3_released.html
____________________ _____________________ __ _____________________________
< server address > < library > <one of the returned results>
#+END_EXAMPLE
*** steps
1. auto start ~kiwix-serve~ HTTP server.
2. query/search on kiwix server.
1. open kiwix server index page to input to search. (But this is slow, waste time)
2. use http language binding library to query on kiwix HTTP server.
1. select library in library list page.
2. after load a library, simulate type query string in the search input
box, the submit to search.
3. return the result page HTML or page URL.
4. view the result with page URL or page HTML with Emacs browser.
*** auto start kiwix HTTP server
Here is a simple script, you can put it in Linux "*auto-start*".
#+begin_src org
,#+BEGIN_SRC sh :tangle "~/scripts/kiwix-server.sh"
#!/usr/bin/env sh
/usr/lib/kiwix/bin/kiwix-serve --library --port=8000 --daemon ~/.www.kiwix.org/kiwix/8ip89lik.default/data/library/library.xml
,#+END_SRC
#+end_src
*** search
1. kiwix-search command -> return a list of results.
#+begin_src org
,#+BEGIN_SRC sh
/usr/lib/kiwix/bin/kiwix-search ~/.www.kiwix.org/kiwix/8ip89lik.default/data/index/wikinews_en_all_2015-11.zim.idx linux
,#+END_SRC
#+end_src
2. use one element of list as part of the URL.
http://127.0.0.1:8000/wikinews_en_all_2015-11/A/Big_Linux_Beta_3_released.html
#+begin_src org
,#+BEGIN_SRC emacs-lisp
(browse-url (concat "http://127.0.0.1:8000/" "LIBRARY" "/A/" "RESULT"))
,#+END_SRC
#+end_src
*** more advanced?
If you want more advanced functions, you can use communicate kiwix HTTP server
with RESTful API.
- I don't know what Emacs library to use.
- Or you can use other language to do this, like Ruby or Python etc.
* Footnotes
[fn:1] https://github.com/kiwix/kiwix-tools/issues/257