Imports rearrangements, readme reformat, basic stuff.

Smoke tests are 100/100.

MANIFEST.in

@@ -1,2 +0,0 @@
-graft ebook_converter/data
-exclude .gitignore

README.rst

@@ -2,19 +2,19 @@
 Ebook converter
 ===============
 
-This is an impudent ripoff of the bits from `Calibre project`_, and is aimed only
+This is an impudent ripoff of the bits from `Calibre project`_, and is aimed
-for converter thing.
+only for converter thing.
 
-My motivation is to have only the converter for ebooks run from the commandline,
+My motivation is to have only the converter for ebooks run from the
-without all of those bells and whistles Calibre has, and with cleanest more
+commandline, without all of those bells and whistles Calibre has, and with
-*pythonic* approach.
+cleanest more *pythonic* approach.
 
 Requirements
 ------------
 
 To build and run ebook converter, you'll need:
 
-- Python 3.6 or newer
+- Python 3.10 or newer
 - `Liberation fonts`_
 - setuptools
 - ``pdftohtml``, ``pdfinfo`` and ``pdftoppm`` from `poppler`_ project for
@@ -22,6 +22,20 @@ To build and run ebook converter, you'll need:
 - ``libxml2-dev`` and ``libxslt-dev`` as dependencies for format manipulation
   from some of the Calibre code
 
+and several Python packages:
+
+- `beautifulsoup4`_
+- `css-parser`_
+- `filelock`_
+- `html2text`_
+- `html5-parser`_
+- `msgpack`_
+- `odfpy`_
+- `pillow`_
+- `python-dateutil`_
+- `setuptools`_
+- `tinycss`_
+
 No Python2 support. Even if Calibre probably still is able to run on Python2, I
 do not have an intention to support it.
 
@@ -29,9 +43,9 @@ do not have an intention to support it.
 What's supported
 ----------------
 
-To be able to perform some optimization and make the converter more reliable and
+To be able to perform some optimization and make the converter more reliable
-easy to use, first I need to remove some of the features, which are totally not
+and easy to use, first I need to remove some of the features, which are totally
-crucial in my opinion, although they might be re-added later, like, for
+not crucial in my opinion, although they might be re-added later, like, for
 instance there is no automatic language translations depending on the locale
 settings.
 
@@ -45,9 +59,10 @@ Windows is not currently supported, because of the original spaghetti code.
 This may change in the future, after cleanup of mentioned pasta would be
 completed.
 
-So called `Kindle periodical` format is not supported, since all we do care are
+So called *Kindle periodical* format (which `Amazon has`_ `killed`_ anyway back
-local files. If there would be downloaded periodical thing (using Calibre for
+in September 2023) is not supported, since all we do care are local files. If
-example), it would be treated as common book.
+there would be downloaded periodical thing (using Calibre for example), it
+would be treated as common book.
 
 
 Input formats
@@ -108,7 +123,7 @@ managers), i.e:
    $ . venv/bin/activate
    (venv) $ git clone https://github.com/gryf/ebook-converter
    (venv) $ cd ebook-converter
-   (venv) $ pip install -r requirements.txt .
+   (venv) $ pip install --no-binary lxml .
 
 Simple as that. And from now on, you can issue converter:
 
@@ -116,6 +131,11 @@ Simple as that. And from now on, you can issue converter:
 
    (venv) $ ebook-converter book.docx book.lrf
 
+.. note:: The ``--no-binary lxml`` flag is required to ensure ``lxml`` is
+   compiled against the same system ``libxml2`` as ``html5-parser``. Without
+   it, pip may install a pre-built ``lxml`` wheel bundling a different
+   ``libxml2`` version, which causes a ``RuntimeError``.
+
 
 License
 -------
@@ -123,9 +143,20 @@ License
 This work is licensed on GPL3 license, like the original work. See LICENSE file
 for details.
 
-
 .. _Calibre project: https://calibre-ebook.com/
 .. _pypi: https://pypi.python.org
 .. _Liberation fonts: https://github.com/liberationfonts/liberation-fonts
-.. _Kindle periodical: https://sellercentral.amazon.com/gp/help/external/help.html?itemID=202047960&language=en-US
+.. _Amazon has: https://goodereader.com/blog/kindle/amazon-will-discontinue-newspaper-and-magazine-subscriptions-in-september
+.. _killed: https://www.theverge.com/23861370/amazon-kindle-periodicals-unlimited-ended
 .. _poppler: https://poppler.freedesktop.org/
+.. _beautifulsoup4: https://www.crummy.com/software/BeautifulSoup
+.. _css-parser: https://github.com/ebook-utils/css-parser
+.. _filelock: https://github.com/tox-dev/py-filelock
+.. _html2text: https://github.com/Alir3z4/html2text
+.. _html5-parser: https://html5-parser.readthedocs.io
+.. _msgpack: https://msgpack.org
+.. _odfpy: https://github.com/eea/odfpy
+.. _pillow: https://python-pillow.github.io
+.. _python-dateutil: https://github.com/dateutil/dateutil
+.. _setuptools: https://setuptools.pypa.io
+.. _tinycss: http://tinycss.readthedocs.io

ebook_converter/__init__.py

@@ -1,6 +1,6 @@
+import importlib.resources
 import mimetypes
-import pkg_resources
 
 
-mimetypes.init([pkg_resources.
+mimetypes.init([str(importlib.resources.
-                resource_filename('ebook_converter', 'data/mime.types')])
+                files('ebook_converter') / 'data/mime.types')])

ebook_converter/ebooks/conversion/cli.py

@@ -2,12 +2,12 @@
 Command line interface to conversion sub-system
 """
 import collections
+import importlib.resources
 import json
 import mimetypes
 import numbers
 import optparse
 import os
-import pkg_resources
 import re
 import sys
 
@@ -351,8 +351,8 @@ def read_sr_patterns(path, log=None):
 
 def main(args=sys.argv):
     log = logging.default_log
-    mimetypes.init([pkg_resources.resource_filename('ebook_converter',
+    mimetypes.init([str(importlib.resources.files('ebook_converter') /
-                                                    'data/mime.types')])
+                        'data/mime.types')])
     parser, plumber = create_option_parser(args, log)
     opts, leftover_args = parser.parse_args(args)
     if len(leftover_args) > 3:

ebook_converter/ebooks/conversion/plugins/fb2_input.py

@@ -1,9 +1,9 @@
 """
 Convert .fb2 files to .lrf
 """
+import importlib.resources
 import mimetypes
 import os
-import pkg_resources
 import re
 
 from lxml import etree
@@ -85,8 +85,8 @@ class FB2Input(InputFormatPlugin):
             css = re.sub(r'name\s*=\s*', 'class=', css)
         self.extract_embedded_content(doc)
         log.debug('Converting XML to HTML...')
-        with open(pkg_resources.resource_filename('ebook_converter',
+        with open(importlib.resources.files('ebook_converter') /
-                                                  'data/fb2.xsl')) as f:
+                  'data/fb2.xsl') as f:
             ss = f.read()
         ss = ss.replace("__FB_NS__", fb_ns)
         if options.no_inline_fb2_toc:

ebook_converter/ebooks/conversion/plugins/html_output.py

@@ -1,5 +1,5 @@
+import importlib.resources
 import os
-import pkg_resources
 import re
 import shutil
 
@@ -92,30 +92,24 @@ class HTMLOutput(OutputFormatPlugin):
             with open(opts.template_html_index, 'rb') as f:
                 template_html_index_data = f.read()
         else:
-            with open(pkg_resources.
+            with open(importlib.resources.files('ebook_converter') /
-                      resource_filename('ebook_converter',
+                      'data/html_export_default_index.tmpl') as fobj:
-                                        'data/html_export_default_index.tmpl')
-                     ) as fobj:
                 template_html_index_data = fobj.read().decode()
 
         if opts.template_html is not None:
             with open(opts.template_html, 'rb') as f:
                 template_html_data = f.read()
         else:
-            with open(pkg_resources.
+            with open(importlib.resources.files('ebook_converter') /
-                      resource_filename('ebook_converter',
+                      'data/html_export_default.tmpl') as fobj:
-                                        'data/html_export_default.tmpl')
-                     ) as fobj:
                 template_html_data = fobj.read().decode()
 
         if opts.template_css is not None:
             with open(opts.template_css, 'rb') as f:
                 template_css_data = f.read()
         else:
-            with open(pkg_resources.
+            with open(importlib.resources.files('ebook_converter') /
-                      resource_filename('ebook_converter',
+                      'data/html_export_default.css') as fobj:
-                                        'data/html_export_default.css')
-                     ) as fobj:
                 template_css_data = fobj.read().decode()
 
         template_html_index_data = template_html_index_data.decode('utf-8')

ebook_converter/ebooks/conversion/plugins/lrf_input.py

@@ -1,6 +1,6 @@
+import importlib.resources
 import os
 import sys
-import pkg_resources
 
 from lxml import etree
 
@@ -52,9 +52,8 @@ class LRFInput(InputFormatPlugin):
 
         self.log.info('Converting XML to HTML...')
 
-        with open(pkg_resources.
+        with open(importlib.resources.files('ebook_converter') /
-                  resource_filename('ebook_converter',
+                  'data/lrf.xsl') as fobj:
-                                    'data/lrf.xsl')) as fobj:
             # TODO(gryf): change this nonsense to etree.parse() instead.
             styledoc = etree.fromstring(fobj.read())
         media_type = MediaType()

ebook_converter/ebooks/conversion/plugins/rtf_input.py

@@ -1,6 +1,6 @@
+import importlib.resources
 import glob
 import os
-import pkg_resources
 import re
 import textwrap
 
@@ -288,8 +288,8 @@ class RTFInput(InputFormatPlugin):
 
         self.log.info('Converting XML to HTML...')
         inline_class = InlineClass(self.log)
-        with open(pkg_resources.resource_filename('ebook_converter',
+        with open(importlib.resources.files('ebook_converter') /
-                                                  'data/rtf.xsl')) as fobj:
+                  'data/rtf.xsl') as fobj:
             styledoc = etree.fromstring(fobj.read())
         extensions = {('calibre', 'inline-class'): inline_class}
         transform = etree.XSLT(styledoc, extensions=extensions)

ebook_converter/ebooks/lrf/pylrs/pylrs.py

@@ -670,7 +670,7 @@ class Info(Delegator):
         f = io.BytesIO()
         tree.write(f, encoding='utf-8', xml_declaration=True)
         xmlInfo = f.getvalue().decode('utf-8')
-        xmlInfo = re.sub(r"<CThumbnail.*?>\n", "", xmlInfo)
+        xmlInfo = re.sub(r"<CThumbnail[^>]*/>\s*", "", xmlInfo)
         xmlInfo = xmlInfo.replace("SumPage>", "Page>")
         lrfWriter.docInfoXml = xmlInfo
 

ebook_converter/ebooks/metadata/opf2.py

@@ -697,7 +697,7 @@ class OPF(object):  # {{{
     def find_toc(self):
         self.toc = None
         try:
-            spine = self.XPath('descendant::*[re:match(name(), "spine", '
+            spine = OPF.XPath('descendant::*[re:match(name(), "spine", '
                                '"i")]')(self.root)
             toc = None
             if spine:
@@ -801,7 +801,7 @@ class OPF(object):  # {{{
 
     def replace_spine_items_by_idref(self, idref, new_idrefs):
         items = list(map(self.create_spine_item, new_idrefs))
-        spine = self.XPath('/opf:package/*[re:match(name(), "spine", '
+        spine = OPF.XPath('/opf:package/*[re:match(name(), "spine", '
                            '"i")]')(self.root)[0]
         old = [i for i in self.iterspine() if i.get('idref', None) == idref]
         for x in old:
@@ -1005,7 +1005,7 @@ class OPF(object):  # {{{
 
     def get_identifiers(self):
         identifiers = {}
-        for x in self.XPath(
+        for x in OPF.XPath(
             'descendant::*[local-name() = "identifier" and text()]')(
                     self.metadata):
             found_scheme = False
@@ -1039,7 +1039,7 @@ class OPF(object):  # {{{
                 uuid_id = self.root.attrib[attr]
                 break
 
-        for x in self.XPath(
+        for x in OPF.XPath(
             'descendant::*[local-name() = "identifier"]')(
                     self.metadata):
             xid = x.get('id', None)
@@ -1175,7 +1175,7 @@ class OPF(object):  # {{{
 
     @property
     def page_progression_direction(self):
-        spine = self.XPath('descendant::*[re:match(name(), "spine", '
+        spine = OPF.XPath('descendant::*[re:match(name(), "spine", '
                            '"i")][1]')(self.root)
         if spine:
             for k, v in spine[0].attrib.items():
@@ -1185,7 +1185,7 @@ class OPF(object):  # {{{
 
     @property
     def primary_writing_mode(self):
-        for m in self.XPath('//*[local-name()="meta" and @name="primary-'
+        for m in OPF.XPath('//*[local-name()="meta" and @name="primary-'
                             'writing-mode" and @content]')(self.root):
             return m.get('content')
 

ebook_converter/ebooks/oeb/polish/toc.py

@@ -1,7 +1,7 @@
 import collections
 import functools
+import importlib.resources
 import operator
-import pkg_resources
 import re
 import urllib.parse
 
@@ -719,9 +719,8 @@ def commit_nav_toc(container, toc, lang=None, landmarks=None,
         if previous_nav is not None:
             root = previous_nav[1]
         else:
-            with open(pkg_resources.
+            with open(importlib.resources.file('ebook_converter') /
-                      resource_filename('ebook_converter',
+                      'data/new_nav.html') as fobj:
-                                        'data/new_nav.html')) as fobj:
                 root = container.parse_xhtml(fobj.read())
         container.replace(tocname, root)
     else:
@@ -874,8 +873,8 @@ def toc_to_html(toc, container, toc_name, title, lang=None):
 
     E = ElementMaker(namespace=const.XHTML_NS, nsmap={None: const.XHTML_NS})
     # TODO(gryf): revisit lack of css.
-    css_f = pkg_resources.resource_filename('ebook_converter',
+    css_f = str(importlib.resources.files('ebook_converter') /
-                                            'data/inline_toc_styles.css')
+                'data/inline_toc_styles.css')
     html = E.html(E.head(E.title(title),
                          E.style(css_f, type='text/css')),
                   E.body(E.h2(title), E.ul(),

ebook_converter/ebooks/oeb/stylizer.py

@@ -1,8 +1,14 @@
 """
 CSS property propagation class.
 """
-import os, re, logging, copy, unicodedata, numbers
+import copy
-import pkg_resources
+import importlib.resources
+import logging
+import numbers
+import os
+import re
+import unicodedata
+
 from operator import itemgetter
 from weakref import WeakKeyDictionary
 from xml.dom import SyntaxErr as CSSSyntaxError
@@ -28,8 +34,8 @@ _html_css_stylesheet = None
 def html_css_stylesheet():
     global _html_css_stylesheet
     if _html_css_stylesheet is None:
-        with open(pkg_resources.resource_filename('ebook_converter',
+        with open(importlib.resources.files('ebook_converter') /
-                                                  'data/html.css'), 'rb') as f:
+                  'data/html.css', 'rb') as f:
             html_css = f.read().decode('utf-8')
         _html_css_stylesheet = parseString(html_css, validate=False)
     return _html_css_stylesheet

ebook_converter/ebooks/oeb/transforms/unsmarten.py

@@ -0,0 +1,28 @@
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+from ebook_converter.ebooks.oeb.base import OEB_DOCS, XPath
+from ebook_converter.ebooks.oeb.parse_utils import barename
+from ebook_converter.utils.unsmarten import unsmarten_text
+
+
+class UnsmartenPunctuation:
+
+    def __init__(self):
+        self.html_tags = XPath('descendant::h:*')
+
+    def unsmarten(self, root):
+        for x in self.html_tags(root):
+            if not barename(x.tag) == 'pre':
+                if getattr(x, 'text', None):
+                    x.text = unsmarten_text(x.text)
+                if getattr(x, 'tail', None) and x.tail:
+                    x.tail = unsmarten_text(x.tail)
+
+    def __call__(self, oeb, context):
+        bx = XPath('//h:body')
+        for x in oeb.manifest.items:
+            if x.media_type in OEB_DOCS:
+                for body in bx(x.data):
+                    self.unsmarten(body)

ebook_converter/ebooks/txt/processor.py

@@ -1,7 +1,8 @@
 """
 Read content from txt file.
 """
-import os, re
+import re
+import os
 
 from ebook_converter.ebooks.metadata.opf2 import OPFCreator
 
@@ -101,7 +102,6 @@ DEFAULT_MD_EXTENSIONS = ('footnotes', 'tables', 'toc')
 
 
 def create_markdown_object(extensions):
-    # Need to load markdown extensions without relying on pkg_resources
     import importlib
     from ebook_converter.ebooks.markdown import Markdown
     from markdown import Extension

ebook_converter/spell/__init__.py

@@ -1,6 +1,6 @@
 from collections import namedtuple
+import importlib.resources
 import json
-import pkg_resources
 
 from ebook_converter.utils.localization import canonicalize_lang
 
@@ -16,9 +16,8 @@ ccodes, ccodemap, country_names = None, None, None
 def get_codes():
     global ccodes, ccodemap, country_names
     if ccodes is None:
-        src = pkg_resources.resource_filename('ebook_converter',
+        with open(importlib.resources.files('ebook_converter') /
-                                              'data/iso_3166-1.json')
+                  'data/iso_3166-1.json', 'rb') as f:
-        with open(src, 'rb') as f:
             db = json.load(f)
         codes = set()
         three_map = {}

ebook_converter/utils/config_base.py

@@ -3,6 +3,7 @@ import collections
 import copy
 import datetime
 import functools
+import importlib.resources
 import json
 import numbers
 import os
@@ -10,8 +11,6 @@ import pickle
 import re
 import traceback
 
-import pkg_resources
-
 from ebook_converter.constants_old import config_dir
 from ebook_converter.constants_old import filesystem_encoding
 from ebook_converter.constants_old import preferred_encoding
@@ -591,8 +590,8 @@ def exec_tweaks(path):
 
 
 def default_tweaks_raw():
-    return pkg_resources.resource_filename('ebook_converter',
+    return str(importlib.resources.files('ebook_converter') /
-                                           'data/default_tweaks.py')
+               'data/default_tweaks.py')
 
 
 def read_tweaks():

ebook_converter/utils/localization.py

@@ -1,5 +1,5 @@
+import importlib.resources
 import json
-import pkg_resources
 
 
 def get_lang():
@@ -39,10 +39,8 @@ def _load_iso639():
     # excerpt form Calibre transform code which is executed during Calibre
     # build).
     if _iso639 is None:
-        src = pkg_resources.resource_filename('ebook_converter',
+        with open(importlib.resources.files('ebook_converter') /
-                                              'data/iso_639-3.json')
+                  'data/iso_639-3.json', 'rb') as f:
-
-        with open(src, 'rb') as f:
             root = json.load(f)
 
         entries = root['639-3']

ebook_converter/utils/unsmarten.py

@@ -0,0 +1,40 @@
+__license__ = 'GPL 3'
+__copyright__ = '2011, John Schember <john@nachtimwald.com>'
+__docformat__ = 'restructuredtext en'
+
+from ebook_converter.utils.mreplace import MReplace
+
+_mreplace = MReplace({
+        '&#8211;': '--',
+        '&ndash;': '--',
+        '–': '--',
+        '&#8212;': '---',
+        '&mdash;': '---',
+        '—': '---',
+        '&#8230;': '...',
+        '&hellip;': '...',
+        '…': '...',
+        '&#8220;': '"',
+        '&#8221;': '"',
+        '&#8222;': '"',
+        '&#8243;': '"',
+        '&ldquo;': '"',
+        '&rdquo;': '"',
+        '&bdquo;': '"',
+        '&Prime;': '"',
+        '“':'"',
+        '”':'"',
+        '„':'"',
+        '″':'"',
+        '&#8216;':"'",
+        '&#8217;':"'",
+        '&#8242;':"'",
+        '&lsquo;':"'",
+        '&rsquo;':"'",
+        '&prime;':"'",
+        '‘':"'",
+        '’':"'",
+        '′':"'",
+})
+
+unsmarten_text = _mreplace.mreplace

pyproject.toml

@@ -1,5 +1,5 @@
 [build-system]
-requires = ["setuptools >= 61.0"]
+requires = ["setuptools >= 77.0"]
 build-backend = "setuptools.build_meta"
 
 [project]
@@ -8,27 +8,26 @@ version = "4.12.0"
 requires-python = ">= 3.10"
 description = "Convert ebook between different formats"
 dependencies = [
-    "filelock",
+    "beautifulsoup4>=4.9.3",
-    "python-dateutil",
+    "css-parser>=1.0.6",
-    "lxml",
+    "filelock>=3.0.12",
-    "css-parser",
+    "html2text>=2020.1.16",
-    "beautifulsoup4",
+    "html5-parser==0.4.12",
-    "tinycss",
+    "msgpack>=1.0.0",
-    "pillow",
+    "odfpy>=1.4.1",
-    "msgpack",
+    "pillow>=8.0.1",
-    "html5-parser",
+    "python-dateutil>=2.8.1",
-    "odfpy",
+    "setuptools>=61.0",
-    "html2text"
+    "tinycss>=0.4"
 ]
 readme = "README.rst"
 authors = [
     {name = "gryf", email = "gryf73@gmail.com"}
 ]
-license = {text = "GPLv3"}
+license = "GPL-3.0-or-later"
 classifiers = [
     "Environment :: Console",
     "Intended Audience :: Other Audience",
-    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
     "Operating System :: POSIX :: Linux",
     "Development Status :: 3 - Alpha",
     "Programming Language :: Python",

requirements.txt

@@ -1,11 +0,0 @@
-beautifulsoup4>=4.9.3
-css-parser>=1.0.6
-filelock>=3.0.12
-html2text>=2020.1.16
-html5-parser==0.4.9 --no-binary lxml
-msgpack>=1.0.0
-odfpy>=1.4.1
-pillow>=8.0.1
-python-dateutil>=2.8.1
-setuptools>=50.3.2
-tinycss>=0.4

setup.cfg

@@ -1,46 +0,0 @@
-[metadata]
-name = ebook-converter
-version = 4.12.0
-summary = Convert ebook between different formats
-description-file =
-    README.rst
-author = gryf
-author-email = gryf73@gmail.com
-license = GPL3
-license_file = LICENSE
-url = https://github.com/gryf/ebook-converter
-classifier =
-    Environment :: Console
-    Intended Audience :: Other Audience
-    License :: OSI Approved :: GNU General Public License v3 (GPLv3)
-    Operating System :: POSIX :: Linux
-    Development Status :: 3 - Alpha
-    Programming Language :: Python
-    Programming Language :: Python :: 3
-    Programming Language :: Python :: 3 :: Only
-    Programming Language :: Python :: 3.6
-    Programming Language :: Python :: 3.7
-
-[options]
-packages = find:
-include_package_data = True
-install_requires =
-    filelock
-    python-dateutil
-    lxml
-    css-parser
-    beautifulsoup4
-    tinycss
-    pillow
-    msgpack
-    html5-parser
-    odfpy
-    setuptools
-    html2text
-
-[options.entry_points]
-console_scripts =
-    ebook-converter=ebook_converter.main:main
-
-[options.package_data]
-* = *.types, *.css, *.html, *.xsl

setup.py

@@ -1,4 +0,0 @@
-import setuptools
-
-
-setuptools.setup()