Adjusted reporters to handle files attached to messages.

Couple of months ago, file attachments was treated differently, than
now - API provides attached files as a list, instead of single object,
and is directly coupled with a message object. This change provides
support for such cases.

.travis.yml

@@ -1,8 +1,6 @@
 language: python
 env:
-- TOXENV=py27
 - TOXENV=py34
-- TOXENV=py27-flake8
 - TOXENV=py34-flake8
 install: pip install tox
 script: tox

MANIFEST.in

@@ -0,0 +1,2 @@
+include README.rst
+include config_example

README.rst

@@ -4,15 +4,19 @@ Slack backup
 .. image:: https://travis-ci.org/gryf/slack-backup.svg?branch=master
     :target: https://travis-ci.org/gryf/slack-backup
 
-This project aim is to collect conversations from Slack using its API and
+.. image:: https://img.shields.io/pypi/v/slack-backup.svg
+    :target: https://pypi.python.org/pypi/slack-backup
+
+The project aim is to collect conversations from Slack using its API and
 optionally user account information, and provides convenient way to represent
 as a log.
 
 Requirements
 ------------
 
-This project is written in Python 2.7, and 3.4+, although version 2.7, which
+This project is written in Python 3, 3.4 to be precise (currently it works with 
-should work, wasn't tested as extensively as it should be.
+version 3.6), although it may work on earlier version of Python3. Sorry no 
+support for Python2.
 
 Other than that, required packages are as follows:
 
@@ -65,7 +69,7 @@ typical session:
    (myenv)user@localhost ~/mylogs $ slack-backup fetch \
    --token xxxx-1111111111-222222222222-333333333333-r4nd0ms7uff \
    --user some@email.address.org --password secret --team myteam \
-   -qqq -d mydatabase.sqlite
+   -qq -d mydatabase.sqlite
 
 where:
 
@@ -87,6 +91,34 @@ where:
   created, but you'll (obviously) lost all the records. Besides the db file,
   assets directory might be created for downloadable items.
 
+You can also specify directory, where pure response JSONs from Slack API will
+be stored by using ``-r/--raw-dir`` or by providing it in config file in
+``fetch`` section as ``raw_dir`` (note the underscore in config file contrary
+to the swith, which have hyphen between ``raw`` and ``dir``). This might be useful for
+debugging purposes.
+
+There is one more switch to take into consideration -
+``-f/--url-file-to-attachment`` which influence the way how external file
+share would be treated. First of all, what is *external* file share from slack
+point of view, one could ask. Slack have some sort of integration with Google
+services, like Google Drive, which provide slack users to create or "upload"
+files from Google Drive. "Upload", since no uploading actually takes place,
+and only URL is provided for such "uploads". By default `slack-backup` will
+create a file which is prefixed ``manual_download_`` which will contain URL and
+destination path to the file, where user should manual download file to.
+Example file contents:
+
+.. code::
+
+   http://foo.bar.com/some/file --> assets/files/83340cbe-fee2-4d2e-bdb1-cace9c82e6d4
+   http://foo.bar.com/some/other/file --> assets/files/8a4c873c-1864-4f1b-b515-bbef119f33a3
+   http://docs/google.com/some/gdoc/file --> assets/files/ec8752bc-0bf8-4743-a8bd-9756107ab386
+
+By setting ``--url-file-to-attachment`` flag (or making an option
+``url_file_to_attachment`` set to ``true`` in config file) such "uploads" would
+be internally converted into Slack "attachment", which internally is an object
+to store external links, so there is no need for user interaction.
+
 During DB creation, all available messages are stored in the database. On the
 next run, ``fetch`` would only take those records, which are older from
 currently oldest in DB. So that it will only fetch a subset of the overall of
@@ -113,6 +145,62 @@ command.
 
 See help for the ``slack-backup`` command for complete list of options.
 
+Configuration
+-------------
+
+For convenience, you can place all of needed options into configuration file
+(aka .ini), which all options (with their defaults) will look like:
+
+.. code:: ini
+
+   [common]
+   channels =
+   database =
+   quiet = 0
+   verbose = 0
+
+   [generate]
+   output =
+   format = text
+   theme = plain
+
+   [fetch]
+   url_file_to_attachment = false
+   user =
+   password =
+   team =
+   token =
+   raw_dir =
+
+Note, that you don't have to put every option. To illustrate ``fetch`` example
+from above, here is a corresponding config file:
+
+.. code:: ini
+
+   [common]
+   database = mydatabase.sqlite
+   quiet = 2
+
+   [fetch]
+   user = some@email.address.org
+   password = secret
+   team = myteam
+   token = xxxx-1111111111-222222222222-333333333333-r4nd0ms7uff
+
+Note, that only ``[common]`` and ``[fetch]`` sections are provided, so it is
+enough to invoke ``slack-backup`` command as:
+
+.. code:: shell-session
+
+   (myenv)user@localhost ~/mylogs $ slack-backup fetch
+
+There are couple of places, where configuration file would be searched for, in
+particular order:
+
+* file provided via argument ``-i`` or ``--config``
+* ``slack-backup.ini`` in current directory
+* ``$XDG_CONFIG_HOME/slack-backup.ini``, where ``$XDG_CONFIG_HOME`` usually
+  defaults to ``$HOME/.config``
 
 Details
 -------

scripts/slack-backup

@@ -1,108 +1,10 @@
 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 """
-Create backup for certain date for specified channel in slack
+Execute commands for slack-backup
 """
-import argparse
+from slack_backup import command
-import logging
-
-from slack_backup import client
-
-
-def setup_logger(args):
-    """Setup logger format and level"""
-
-    level = logging.WARNING
-
-    if args.quiet:
-        level = logging.ERROR
-        if args.quiet > 1:
-            level = logging.CRITICAL
-
-    if args.verbose:
-        level = logging.INFO
-        if args.verbose > 1:
-            level = logging.DEBUG
-
-    logging.basicConfig(level=level,
-                        format="%(asctime)s %(levelname)s: %(message)s")
-
-def generate_raport(args):
-    """Generate logs"""
-    slack = client.Client(args)
-    slack.generate_history()
-
-def fetch_data(args):
-    """Fetch and store data"""
-    slack = client.Client(args)
-    slack.update()
-
-
-def main():
-    """Main function"""
-    parser = argparse.ArgumentParser()
-
-    subparser = parser.add_subparsers(dest='parser')
-    subparser.required = True
-
-    fetch = subparser.add_parser('fetch', help='Update local db with Slack'
-                                 ' data')
-    fetch.add_argument('-t', '--token', required=True, help='Slack token - '
-                       'a string, which can be generated/obtained via '
-                       'https://api.slack.com/docs/oauth-test-tokens page.')
-    fetch.add_argument('-u', '--user', default='', help='Username for your '
-                       'Slack account')
-    fetch.add_argument('-p', '--password', default='', help='Password for your '
-                       'Slack account.')
-    fetch.add_argument('-e', '--team', required=True, help='Team name, which '
-                       'is part of slack url, for example: if url is '
-                       '"https://team.slack.com" than "team" is a name of '
-                       'the team.')
-    fetch.add_argument('-v', '--verbose', help='Be verbose. Adding more "v"'
-                       'will increase verbosity', action="count", default=0)
-    fetch.add_argument('-q', '--quiet', help='Be quiet. Adding more "q"'
-                       'will decrease verbosity', action="count", default=0)
-    fetch.add_argument('-c', '--channels', default=[], nargs='+',
-                       help='List of channels to perform actions on. '
-                       'Default is all channels.')
-
-    fetch.add_argument('-d', '--database', default='',
-                       help='Path to the database file.')
-
-    fetch.set_defaults(func=fetch_data)
-
-    generate = subparser.add_parser('generate', help='Generate logs out of '
-                                    'data in provided database')
-    generate.add_argument('-o', '--output', default='logs', help="Output "
-                          "directory for store logs. All logs are organised "
-                          "per channel. By default it's `logs' directory")
-    generate.add_argument('-f', '--format', default='none',
-                          choices=('text', 'none'),
-                          help='Output format. Default is none; only database '
-                          'is updated by latest messages for all/selected '
-                          'channels.')
-    generate.add_argument('-t', '--theme', default='plain',
-                          choices=('plain', 'unicode'),
-                          help='Choose theme for text output. It doesn\'t '
-                          'affect other output formats.')
-
-    generate.add_argument('-v', '--verbose', help='Be verbose. Adding more "v"'
-                          'will increase verbosity', action="count", default=0)
-    generate.add_argument('-q', '--quiet', help='Be quiet. Adding more "q"'
-                          'will decrease verbosity', action="count", default=0)
-    generate.add_argument('-c', '--channels', default=[], nargs='+',
-                          help='List of channels to perform actions on. '
-                          'Default is all channels.')
-
-    generate.add_argument('-d', '--database', default='',
-                          help='Path to the database file.')
-
-    generate.set_defaults(func=generate_raport)
-
-    args = parser.parse_args()
-    setup_logger(args)
-    args.func(args)
 
 
 if __name__ == "__main__":
-    main()
+    command.main()

setup.py

@@ -2,12 +2,15 @@
 """
 Setup for the slack-backup project
 """
-from distutils.core import setup
+try:
+    from setuptools import setup
+except ImportError:
+    from distutils.core import setup
 
 
 setup(name="slack-backup",
       packages=["slack_backup"],
-      version="0.2",
+      version="0.8",
       description="Make copy of slack converstaions",
       author="Roman Dobosz",
       author_email="gryf73@gmail.com",
@@ -18,7 +21,7 @@ setup(name="slack-backup",
       scripts=["scripts/slack-backup"],
       classifiers=["Programming Language :: Python :: 3",
                    "Programming Language :: Python :: 3.4",
-                   "Development Status :: 3 - Alpha",
+                   "Development Status :: 4 - Beta",
                    "Environment :: Console",
                    "Intended Audience :: End Users/Desktop",
                    "License :: OSI Approved :: BSD License",

slack_backup/client.py

@@ -3,15 +3,20 @@ Create backup for certain date for specified channel in slack
 """
 from datetime import datetime
 import getpass
+import json
 import logging
 import os
+import pprint
+import uuid
 
 import slackclient
+import sqlalchemy.orm.exc
 
 from slack_backup import db
 from slack_backup import objects as o
 from slack_backup import download
 from slack_backup import reporters
+from slack_backup import utils
 
 
 class Client(object):
@@ -19,6 +24,8 @@ class Client(object):
     This class is intended to provide an interface for getting, storing and
     querying data fetched out using Slack API.
     """
+    RAW = '%Y%m%d%H%M%S_{name}.json'
+
     def __init__(self, args):
         if 'token' in args:
             self.slack = slackclient.SlackClient(args.token)
@@ -40,9 +47,24 @@ class Client(object):
         self.selected_channels = args.channels
         self.q = self.session.query
 
+        self._raw_fname = None
+        if 'raw_dir' in args and args.raw_dir:
+            if not os.path.exists(args.raw_dir):
+                os.mkdir(args.raw_dir)
+            fpath = os.path.join(args.raw_dir, self.RAW)
+            self._raw_fname = datetime.now().strftime(fpath)
+
         if 'format' in args:
             self.reporter = reporters.get_reporter(args, self.q)
 
+        if 'url_file_to_attachment' in args:
+            self._url_file_to_attachment = args.url_file_to_attachment
+
+        self._dlpath = utils.get_temp_name(dir=os.path.curdir,
+                                           prefix='manual_download_',
+                                           unlink=True)
+        self._dldata = []
+
     def update(self):
         """
         Perform an update, store data to db
@@ -51,6 +73,7 @@ class Client(object):
         self.update_users()
         self.update_channels()
         self.update_history()
+        self._finalize()
 
     def update_channels(self):
         """Fetch and update channel list with current state in db"""
@@ -60,6 +83,10 @@ class Client(object):
         if not result:
             return
 
+        if self._raw_fname:
+            with open(self._raw_fname.format(name='channels'), 'w') as fobj:
+                fobj.write(json.dumps(result))
+
         for data in result:
             channel = self.q(o.Channel).\
                 filter(o.Channel.slackid == data['id']).one_or_none()
@@ -74,14 +101,16 @@ class Client(object):
 
     def update_users(self):
         """Fetch and update user list with current state in db"""
-        logging.info("Fetching and updating user information in DB")
+        result = self._users_list()
-        result = self.slack.api_call("users.list", presence=0)
 
-        if not result.get("ok"):
+        if not result:
-            logging.error(result['error'])
             return
 
-        for user_data in result['members']:
+        if self._raw_fname:
+            with open(self._raw_fname.format(name='users'), 'w') as fobj:
+                fobj.write(json.dumps(result))
+
+        for user_data in result:
             user = self.q(o.User).\
                 filter(o.User.slackid == user_data['id']).one_or_none()
 
@@ -112,6 +141,7 @@ class Client(object):
             channels = all_channels
 
         for channel in channels:
+            logging.info("Getting messages for channel `%s'", channel.name)
             latest = self.q(o.Message).\
                 filter(o.Message.channel == channel).\
                 order_by(o.Message.ts.desc()).first()
@@ -123,9 +153,15 @@ class Client(object):
             # starting from first January 1970.
             latest = latest and latest.ts or 1
 
+            result = []
             while True:
                 logging.debug("Fetching another portion of messages")
                 messages, latest = self._channels_history(channel, latest)
+                if messages is None:
+                    # ignore deleted channels
+                    break
+
+                result.extend(messages)
 
                 for msg in messages:
                     self._create_message(msg, channel)
@@ -133,6 +169,11 @@ class Client(object):
                 if latest is None:
                     break
 
+            if self._raw_fname:
+                with open(self._raw_fname.format(name='channel-' +
+                                                 channel.name), 'w') as fobj:
+                    fobj.write(json.dumps(result))
+
         self.session.commit()
 
     def generate_history(self):
@@ -141,15 +182,64 @@ class Client(object):
         """
         self.reporter.generate()
 
+    def _get_user(self, data):
+        """
+        Return an User object. It can be regular one, or a bot. In case of
+        bot, check if it exists in db, and in case of failure - create it,
+        since bots are not returned by user.list API method.
+        """
+        try:
+            return self.q(o.User).filter(o.User.slackid == data['user']).one()
+        except KeyError:
+            pass
+
+        try:
+            return self.q(o.User).filter(o.User.slackid ==
+                                         data['comment']['user']).one()
+        except KeyError:
+            pass
+
+        try:
+            return self.q(o.User).filter(o.User.slackid ==
+                                         data['bot_id']).one()
+        except KeyError:
+            pass
+        except sqlalchemy.orm.exc.NoResultFound:
+            result = self.slack.api_call('bots.info', bot=data['bot_id'])
+            if not result.get("ok"):
+                logging.error(result['error'])
+                return None
+
+            user = o.User(result['bot'])
+            user.real_name = result['bot']['name']
+            self.session.add(user)
+            self.session.flush()
+
+            if self._raw_fname:
+                with open(self._raw_fname.format(name='bot-' + user.slackid),
+                          "w") as fobj:
+                    fobj.write(json.dumps(result))
+
+            return user
+
+        logging.exception('Failed on data: %s', pprint.pformat(data))
+        raise ValueError('Cannot identify user out of given data.')
+
     def _create_message(self, data, channel):
         """
         Create message with corresponding possible metadata, like reactions,
         files etc.
         """
-        user = self.q(o.User).\
+        if data['type'] != 'message':
-            filter(o.User.slackid == data['user']).one()
+            logging.info("Skipping message of type `%s'.", data['type'])
+            return
 
-        if data['type'] == 'message' and not data['text'].strip():
+        logging.debug('Message data: %s', json.dumps(data))
+
+        user = self._get_user(data)
+
+        if not any((data.get('attachments'), data['text'].strip(),
+                    data.get('files'))):
             logging.info("Skipping message from `%s' since it's empty",
                          user.name)
             return
@@ -165,37 +255,67 @@ class Client(object):
             for reaction_data in data['reactions']:
                 message.reactions.append(o.Reaction(reaction_data))
 
-        if data.get('subtype') == 'file_share':
+        if data.get('files'):
-            self._file_data(message, data['file'], data['file']['is_external'])
+            for fdata in data['files']:
-        elif data.get('subtype') == 'pinned_item':
+                if (self._url_file_to_attachment and fdata.get('is_external')):
+                    logging.info('got external file')
+                    message.text = (message.text.split('shared a file:')[0] +
+                                    'shared a file: ')
+                    logging.debug("Found external file `%s'. Saving as "
+                                  "attachment.", fdata['url_private'])
+                    self._att_data(message, [{'title': fdata['name'],
+                                              'text': fdata['url_private'],
+                                              'fallback': ''}])
+                else:
+                    self._file_data(message, fdata)
+
+        # TODO(gryf): subtype pinned_item coexistsing with pinned_info message
+        # key :C
+        # pinned_info however is just a mark, which point to the channlel
+        # where it is pinned to, who did that and when. To be resolved.
+        if data.get('subtype') == 'pinned_item':
             if data.get('attachments'):
                 self._att_data(message, data['attachments'])
             elif data.get('item'):
-                self._file_data(message, data['item'],
+                self._file_data(message, data['item'])
-                                data['item']['is_external'])
         elif data.get('attachments'):
             self._att_data(message, data['attachments'])
 
         self.session.add(message)
 
-    def _file_data(self, message, data, is_external=True):
+    def _file_data(self, message, data):
         """
         Process file data. Could be either represented as 'file' object or
         'item' object in case of pinned items
         """
-        message.file = o.File(data)
+        _file = o.File(data)
+        message.files.append(_file)
+
+        if data.get('mode') == 'tombstone':
+            _file.title = 'This file was deleted'
+            return
+
         if data.get('is_starred'):
             message.is_starred = True
 
-        if is_external:
+        if data.get('is_external'):
-            logging.debug("Found external file `%s'", data['url_private'])
+            # Create a link and corresponding file name for manual download
-            message.file.url = data['url_private']
+            fname = str(uuid.uuid4())
+            _file.filepath = self.downloader.get_filepath(fname, 'file')
+            logging.info("Please, manually download an external file from "
+                         "URL `%s' to `%s'", data['url_private'],
+                         _file.filepath)
+            self._dldata.append('%s --> %s\n' % (data['url_private'],
+                                                 _file.filepath))
+            _file.url = data['url_private']
         else:
             logging.debug("Found internal file `%s'",
                           data['url_private_download'])
             priv_url = data['url_private_download']
-            message.file.filepath = self.downloader.download(priv_url, 'file')
+            _file.filepath = self.downloader.download(priv_url, 'file',
-        self.session.add(message.file)
+                                                      data.get('filetype'))
+
+        self.session.add(_file)
 
     def _att_data(self, message, data):
         """
@@ -216,11 +336,11 @@ class Client(object):
         user = self.q(o.User).filter(o.User.slackid ==
                                      data['creator']).one_or_none()
 
-        obj = self.q(classobj).\
+        obj = (self.q(classobj).
-            filter(classobj.last_set ==
+               filter(classobj.last_set ==
-                   datetime.fromtimestamp(data['last_set'])).\
+                      utils.fromtimestamp(data['last_set'])).
-            filter(classobj.value == data['value']).\
+               filter(classobj.value == data['value']).
-            filter(classobj.creator == user).one_or_none()
+               filter(classobj.creator == user).one_or_none())
 
         if not obj:
             # break channel relation
@@ -255,7 +375,7 @@ class Client(object):
         if not database:
             return 'assets'
 
-        path = os.path.dirname(os.path.abspath(database))
+        path = os.path.dirname(database)
         return os.path.join(path, 'assets')
 
     def _channels_list(self):
@@ -276,7 +396,8 @@ class Client(object):
         Get users list using Slack API. Return list of channel data or None
         in case of error.
         """
-        result = self.slack.api_call("users.list", presence=0)
+        logging.info("Fetching and updating user information in DB")
+        result = self.slack.api_call("users.list")
 
         if not result.get("ok"):
             logging.error(result['error'])
@@ -309,3 +430,14 @@ class Client(object):
                 return result['messages'], None
 
         return [], None
+
+    def _finalize(self):
+        """Create misc files if necessary - like manual donwload"""
+        if not self._dldata:
+            return
+
+        with open(self._dlpath, "a") as fobj:
+            fobj.write(''.join(self._dldata))
+        logging.warning("Manual action required! Download all the files "
+                        "listed in `%s' and each of them save as file listed "
+                        "right after `-->' sign", self._dlpath)

slack_backup/command.py

@@ -0,0 +1,143 @@
+"""
+Create backup for certain date for specified channel in slack
+"""
+import argparse
+import logging
+import platform
+
+from slack_backup import client
+from slack_backup import config
+
+
+def setup_logger(args):
+    """Setup logger format and level"""
+
+    if platform.system() != "Windows":
+        # hack to have colors in terminal
+        logging.addLevelName(logging.DEBUG,
+                             "\033[1;30m%s\033[1;0m" %
+                             logging.getLevelName(logging.DEBUG))
+
+        logging.addLevelName(logging.INFO,
+                             "\033[1;32m%s\033[1;0m" %
+                             logging.getLevelName(logging.INFO))
+
+        logging.addLevelName(logging.WARNING,
+                             "\033[1;33m%s\033[1;0m" %
+                             logging.getLevelName(logging.WARNING))
+
+        logging.addLevelName(logging.ERROR,
+                             "\033[1;31m%s\033[1;0m" %
+                             logging.getLevelName(logging.ERROR))
+
+        logging.addLevelName(logging.CRITICAL,
+                             "\033[7;31m%s\033[1;0m" %
+                             logging.getLevelName(logging.CRITICAL))
+
+    level = logging.WARNING
+
+    if args.quiet:
+        level = logging.ERROR
+        if args.quiet > 1:
+            level = logging.CRITICAL
+
+    if args.verbose:
+        level = logging.INFO
+        if args.verbose > 1:
+            level = logging.DEBUG
+
+    logging.basicConfig(level=level,
+                        format="%(asctime)s %(levelname)s: %(message)s")
+
+
+def generate_raport(args):
+    """Generate logs"""
+    slack = client.Client(args)
+    slack.generate_history()
+
+
+def fetch_data(args):
+    """Fetch and store data"""
+    slack = client.Client(args)
+    slack.update()
+
+
+def main():
+    """Main function"""
+    parser = argparse.ArgumentParser()
+
+    subparser = parser.add_subparsers(dest='parser')
+    subparser.required = True
+
+    fetch = subparser.add_parser('fetch', help='Update local db with Slack'
+                                 ' data')
+    fetch.add_argument('-t', '--token', default=None, help='Slack token - '
+                       'a string, which can be generated/obtained via '
+                       'https://api.slack.com/docs/oauth-test-tokens page.')
+    fetch.add_argument('-u', '--user', default=None, help='Username for your '
+                       'Slack account')
+    fetch.add_argument('-p', '--password', default=None, help='Password for '
+                       'your Slack account.')
+    fetch.add_argument('-e', '--team', default=None, help='Team name, which '
+                       'is part of slack url, for example: if url is '
+                       '"https://team.slack.com" than "team" is a name of '
+                       'the team.')
+    fetch.add_argument('-v', '--verbose', help='Be verbose. Adding more "v" '
+                       'will increase verbosity', action="count",
+                       default=None)
+    fetch.add_argument('-q', '--quiet', help='Be quiet. Adding more "q" will'
+                       ' decrease verbosity', action="count", default=None)
+    fetch.add_argument('-c', '--channels', default=None, nargs='+',
+                       help='List of channels to perform actions on. '
+                       'Default is all channels.')
+    fetch.add_argument('-d', '--database', default=None,
+                       help='Path to the database file.')
+    fetch.add_argument('-i', '--config', default=None,
+                       help='Use specific config file.')
+    fetch.add_argument('-r', '--raw-dir', default=None,
+                       help='Write raw responses to provided directory.')
+    fetch.add_argument('-f', '--url-file-to-attachment', default=False,
+                       action='store_true',
+                       help='Treat shared files (but not uploaded to the '
+                       'Slack servers) as attachment. By default there will '
+                       'be file created in current directory with url and '
+                       'path to the filename under which it would be '
+                       'registered in the DB.')
+    fetch.set_defaults(func=fetch_data)
+
+    generate = subparser.add_parser('generate', help='Generate logs out of '
+                                    'data in provided database')
+    generate.add_argument('-o', '--output', default=None, help="Output "
+                          "directory for store logs. All logs are organised "
+                          "per channel. By default it's `logs' directory")
+    generate.add_argument('-f', '--format', default=None,
+                          choices=('text', 'html', 'none'),
+                          help='Output format. Default is none; only database '
+                          'is updated by latest messages for all/selected '
+                          'channels.')
+    generate.add_argument('-t', '--theme', default=None,
+                          choices=('plain', 'unicode'),
+                          help='Choose theme for text output. It doesn\'t '
+                          'affect other output formats.')
+    generate.add_argument('-v', '--verbose', help='Be verbose. Adding more '
+                          '"v" will increase verbosity', action="count",
+                          default=None)
+    generate.add_argument('-q', '--quiet', help='Be quiet. Adding more "q" '
+                          'will decrease verbosity', action="count",
+                          default=None)
+    generate.add_argument('-c', '--channels', default=[], nargs='+',
+                          help='List of channels to perform actions on. '
+                          'Default is all channels.')
+    generate.add_argument('-d', '--database', default=None,
+                          help='Path to the database file.')
+    generate.add_argument('-i', '--config', default=None,
+                          help='Use specific config file.')
+    generate.set_defaults(func=generate_raport)
+
+    args = parser.parse_args()
+    cfg = config.Config()
+    msg = cfg.update(args)
+    setup_logger(args)
+    logging.info(msg)
+
+    args.func(args)

slack_backup/config.py

@@ -0,0 +1,118 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+"""
+Configuration module for slack-backup
+"""
+import json
+import os
+import configparser
+
+
+class Config(object):
+    """Configuration keeper"""
+
+    ints = ['verbose', 'quiet']
+    bools = ['url_file_to_attachment']
+
+    sections = {'common': ['channels', 'database', 'quiet', 'verbose'],
+                'fetch': ['user', 'password', 'team', 'token',
+                          'url_file_to_attachment', 'raw_dir'],
+                'generate': ['output', 'format', 'theme']}
+
+    def __init__(self):
+        """
+        Init. Read config, if exists, and update passed argument parser
+        object.
+        """
+
+        self.cp = configparser.ConfigParser()
+        self._options = {'channels': [],
+                         'database': None,
+                         'quiet': 0,
+                         'verbose': 0,
+                         'user': None,
+                         'password': None,
+                         'team': None,
+                         'token': None,
+                         'output': None,
+                         'format': None,
+                         'theme': None,
+                         'url_file_to_attachment': False,
+                         'raw_dir': None}
+        # This message supposed to be displayed in INFO level. During the time
+        # of running the code where it should be displayed there is no
+        # complete information about logging level. Displaying message is
+        # dependent on the a) config file, b) argument from commandline. Let's
+        # resolve if user want to have that information or not after merging
+        # those two sources. If user do not want to see any message in INFO
+        # level, we shouldn't do so.
+        self.msg = ''
+
+    def update(self, args):
+        self.load_config(args)
+        self.parse_loaded_options()
+        self.update_args(args)
+        return self.msg
+
+    def load_config(self, args):
+
+        path = ''
+        if hasattr(args, 'config') and args.config:
+            path = args.config
+
+        locations = [path,
+                     './slack-backup.ini',
+                     os.path.expandvars('$XDG_CONFIG_HOME/slack-backup.ini'),
+                     os.path.expandvars('$HOME/.config/slack-backup.ini')]
+
+        for location in locations:
+            if os.path.exists(location):
+                self.cp.read(location)
+                self.msg = 'Found configuration file: %s' % location
+                break
+        else:
+            self.msg = 'No configuration file found'
+
+    def parse_loaded_options(self):
+
+        for section in self.cp.sections():
+            if section not in self.sections:
+                continue
+
+            for option in self.sections[section]:
+                if option in self.ints:
+                    val = self.cp.getint(section, option, fallback=0)
+                elif option in self.bools:
+                    val = self.cp.getboolean(section, option, fallback=False)
+                elif option == 'channels':
+                    val = self.cp.get(section, option, fallback='[]')
+                    val = json.loads(val)
+                else:
+                    val = self.cp.get(section, option, fallback=None)
+
+                self._options[option] = val
+
+    def update_args(self, args):
+        if 'parser' not in args:
+            # it doesn't make sense to update args, since no action was
+            # choosen
+            return
+
+        # special case, re-set information for verbose/quiet options
+        if 'verbose' in args and args.verbose is not None:
+            self._options['verbose'] = args.verbose
+            if self._options['quiet'] is not None:
+                self._options['quiet'] = 0
+        if 'quiet' in args and args.quiet is not None:
+            self._options['quiet'] = args.quiet
+            if self._options['verbose'] is not None:
+                self._options['verbose'] = 0
+
+        for sec_id in (args.parser, 'common'):
+            for option in self.sections[sec_id]:
+                if option in args:
+                    if getattr(args, option) is not None:
+                        continue
+
+                setattr(args, option, self._options[option])

slack_backup/download.py

@@ -2,14 +2,45 @@
 Module for download files, store them in local filesystem and convert the URLs
 to local ones, so that sophisticated writers can make a use of it
 """
+import functools
 import logging
 import os
+import shutil
 
 import requests
 
 from slack_backup import utils
 
 
+def retry(count):
+    """
+    Decorator for a case, when there is some network hiccup, or slack servers
+    are too busy to respond or on connection timeout. Parameter count says how
+    many times it should try to perform request.
+    """
+    def wrapper(func):
+        @functools.wraps(func)
+        def inner(obj, *args, **kwargs):
+            counter = count
+
+            while counter:
+                counter -= 1
+                try:
+                    return func(obj, *args, **kwargs)
+                except requests.exceptions.RequestException as exc:
+                    if not counter:
+                        logging.error('Request for %s failed. Reported '
+                                      'reason: %s', args[0], exc.__doc__)
+                        raise
+                    logging.warning('Request for %s failed. Reported '
+                                    'reason: %s. Retrying.', args[0],
+                                    exc.__doc__)
+                    # Renew the session before retry
+                    obj.authorize()
+        return inner
+    return wrapper
+
+
 class NotAuthorizedError(requests.HTTPError):
     pass
 
@@ -18,7 +49,7 @@ class Download(object):
     """Download class for taking care of Slack internally uploaded files"""
 
     def __init__(self, args, assets_dir):
-        self.session = requests.session()
+        self.session = None
         self.team = args.team
         self.user = args.user
         self.password = args.password
@@ -29,7 +60,7 @@ class Download(object):
         self._hier_created = False
         self.cookies = {}
 
-    def download(self, url, filetype):
+    def download(self, url, filetype, ext=None):
         """
         Download asset, return local path to it
         """
@@ -37,10 +68,26 @@ class Download(object):
         if not self._hier_created:
             self._create_assets_dir()
 
-        fname = self.prepare_filepath(url, filetype)
+        filepath = self.get_filepath(url, filetype, ext)
+        temp_file = utils.get_temp_name()
 
-        self._download(url, fname)
+        self._download(url, temp_file)
-        return fname
+
+        if filepath and os.path.exists(filepath) and filetype != 'avatar':
+            if not utils.same_files(filepath, temp_file):
+                logging.warning("File `%s' already exist, renamed to `%s'",
+                                filepath,
+                                self.calculate_new_filename(filepath,
+                                                            filetype))
+                filepath = self.calculate_new_filename(filepath, filetype)
+                shutil.move(temp_file, filepath)
+            else:
+                logging.debug("File `%s' already exist, skipping", filepath)
+                os.unlink(filepath)
+        else:
+            shutil.move(temp_file, filepath)
+
+        return filepath
 
     def _create_assets_dir(self):
         for path in (self._files, self._images):
@@ -48,20 +95,21 @@ class Download(object):
 
         self._hier_created = True
 
-    def prepare_filepath(self, url, filetype):
+    def get_filepath(self, url, filetype, ext=None):
-        """Prepare directory where to download file into"""
+        """Get full path and filename for the file"""
 
         typemap = {'avatar': self._images,
                    'file': self._files}
 
         if filetype == 'file' and not self._authorized:
-            logging.info("There was no (valid) credentials passed, therefore "
+            logging.warning("There was no (valid) credentials passed, "
-                         "file `%s' cannot be downloaded", url)
+                            "therefore file `%s' cannot be downloaded", url)
             return
 
         splitted = url.split('/')
 
-        if len(splitted) == 7 and 'slack.com' in splitted[2]:
+        if len(splitted) == 7 and ('slack.com' in splitted[2] or
+                                   'slack-edge.com' in splitted[2]):
             part = url.split('/')[-3]
             fname = url.split('/')[-1]
         else:
@@ -75,24 +123,27 @@ class Download(object):
             utils.makedirs(os.path.join(path, part))
             path = os.path.join(path, part)
 
-        path = os.path.join(path, fname)
+        if ext and not fname.endswith(ext):
+            fname = fname + "." + ext
+
+        return os.path.join(path, fname)
+
+    def calculate_new_filename(self, path, filetype):
         count = 1
 
         while filetype != 'avatar' and os.path.exists(path):
-            base, ext = os.path.splitext(path)
+            if count == 1:
-            path = base + "%0.3d" % count + ext
+                base, ext = os.path.splitext(path)
+            path = base + ".%0.3d" % count + ext
+            count += 1
 
         return path
 
+    @retry(3)
     def _download(self, url, local):
         """Download file"""
 
-        try:
+        res = self.session.get(url, stream=True)
-            res = self.session.get(url, stream=True)
-        except requests.exceptions.RequestException as exc:
-            logging.error('Request for %s failed. Reported reason: %s',
-                          url, exc.__doc__)
-            raise
 
         with open(local, 'wb') as fobj:
             for chunk in res.iter_content(chunk_size=5120):
@@ -104,7 +155,9 @@ class Download(object):
         """
         Authenticate and gather session for Slack
         """
+        self.session = requests.session()  # new session
         res = self.session.get('https://%s.slack.com/' % self.team)
+
         if not all((self.team, self.password, self.user)):
             logging.warning('There is neither username, password or team name'
                             ' provided. Downloading will not be performed.')
@@ -112,7 +165,7 @@ class Download(object):
 
         crumb = ''
         for line in res.text.split('\n'):
-            if 'crumb' in line:
+            if 'crumb' in line and 'value' in line:
                 crumb = line.split('value=')[1].split('"')[1]
                 break
         else:
@@ -125,8 +178,7 @@ class Download(object):
                                  'password': self.password,
                                  'signin': 1})
         self.cookies = requests.utils.dict_from_cookiejar(self.session.cookies)
-        if not ('a' in self.cookies and 'b' in self.cookies and
+        if not ('d' in self.cookies and 'd-s' in self.cookies):
-                ('a-' + self.cookies['a']) in self.cookies):
             logging.error('Failed to login into Slack app')
         else:
             self._authorized = True

slack_backup/emoji.py

@@ -1,4 +1,3 @@
-# -*- coding: utf-8 -*-
 """
 This module contains emoji list
 """
@@ -54,7 +53,7 @@ EMOJI = {'plain': {":bowtie:": ':)8',
                    ":scream:": 'X-O',
                    ":tired_face:": 'D-X',
                    ":angry:": "(`##')",
-                   ":rage:": "\(`###')>",
+                   ":rage:": "\\(`###')>",
                    ":triumph:": '(-A-)',
                    ":sleepy:": '(-_-) zZ',
                    ":sunglasses:": 'B-)',

slack_backup/objects.py

@@ -1,13 +1,12 @@
 """
 Convinient object mapping from slack API reponses
 """
-from datetime import datetime
-
 from sqlalchemy import Column, Integer, Text, Boolean, ForeignKey
 from sqlalchemy import DateTime
 from sqlalchemy.orm import relationship
 
 from slack_backup.db import Base
+from slack_backup import utils
 
 
 class Purpose(Base):
@@ -28,7 +27,7 @@ class Purpose(Base):
 
     def update(self, data_dict):
         data_dict = data_dict or {}
-        self.last_set = datetime.fromtimestamp(data_dict.get('last_set', 0))
+        self.last_set = utils.fromtimestamp(data_dict.get('last_set'))
         self.value = data_dict.get('value')
 
     def __repr__(self):
@@ -56,7 +55,7 @@ class Topic(Base):
 
     def update(self, data_dict):
         data_dict = data_dict or {}
-        self.last_set = datetime.fromtimestamp(data_dict.get('last_set', 0))
+        self.last_set = utils.fromtimestamp(data_dict.get('last_set'))
         self.value = data_dict.get('value')
 
     def __repr__(self):
@@ -91,7 +90,7 @@ class Channel(Base):
 
         self.slackid = data_dict.get('id', '')
         self.name = data_dict.get('name', '')
-        self.created = datetime.fromtimestamp(data_dict.get('created', 0))
+        self.created = utils.fromtimestamp(data_dict.get('created'))
         self.is_archived = data_dict.get('is_archived', False)
 
     def __repr__(self):
@@ -207,7 +206,7 @@ class Message(Base):
     __tablename__ = "messages"
 
     id = Column(Integer, primary_key=True)
-    # NOTE(gryf): timestamp from messages are coming as text. It might be
+    # NOTE(gryf): timestamps from messages are coming as text. It might be
     # tempting to store them as Decimal or Integer, but it doesn't really
     # matters since in case of Decimal sqlite doesn't support it, and Integer
     # require additional conversion. It might be critical for messages to have
@@ -229,14 +228,14 @@ class Message(Base):
     channel = relationship("Channel", back_populates="messages")
 
     reactions = relationship("Reaction", back_populates="message")
-    file = relationship("File", uselist=False, back_populates="message")
+    files = relationship("File", back_populates="message")
     attachments = relationship("Attachment", back_populates="message")
 
     def __init__(self, data_dict=None):
         self.update(data_dict)
 
     def datetime(self):
-        return datetime.fromtimestamp(float(self.ts))
+        return utils.fromtimestamp(float(self.ts))
 
     def update(self, data_dict):
         data_dict = data_dict or {}
@@ -256,7 +255,7 @@ class File(Base):
     filepath = Column(Text)
 
     message_id = Column(Integer, ForeignKey('messages.id'))
-    message = relationship('Message', back_populates='file')
+    message = relationship('Message', back_populates='files')
 
     def __init__(self, data_dict=None):
         self.update(data_dict)

slack_backup/reporters.py

@@ -1,19 +1,15 @@
-# -*- coding: utf-8 -*-
 """
 Reporters module.
 
 There are several classes for specific format reporting, and also some of the
 slack conversation/convention parsers.
 """
-from __future__ import absolute_import, division, print_function
 import os
 import errno
+import html.parser
 import logging
+import pathlib
 import re
-try:
-    from html.parser import HTMLParser
-except ImportError:
-    from HTMLParser import HTMLParser
 
 from slack_backup import objects as o
 from slack_backup import utils
@@ -23,6 +19,24 @@ from slack_backup import emoji
 class Reporter(object):
     """Base reporter class"""
     ext = ''
+    symbols = {'plain': {'join': '->',
+                         'leave': '<-',
+                         'me': '*',
+                         'file': '-',
+                         'topic': '+',
+                         'separator': '|'},
+               'unicode': {'join': '⮊',
+                           'leave': '⮈',
+                           'me': '🟊',
+                           'file': '📂',
+                           'topic': '🟅',
+                           'separator': '│'}}
+    literal_url_pat = re.compile(r'(?P<replace>(?P<url>https?[^\s\|]+))')
+    url_pat = re.compile(r'(?P<replace><(?P<url>http[^\|>]+)'
+                         r'(\|(?P<title>[^>]+))?>)')
+    url2_pat = re.compile(r'<(?P<url>https?[^\s\|]+)>')
+    slackid_pat = re.compile(r'(?P<replace><@'
+                             r'(?P<slackid>U[A-Z,0-9]+)(\|[^>]+)?[^>]*>)')
 
     def __init__(self, args, query):
         self.out = args.output
@@ -31,44 +45,47 @@ class Reporter(object):
         self.types = {"channel_join": self._msg_join,
                       "channel_leave": self._msg_leave,
                       "channel_topic": self._msg_topic,
-                      "file_share": self._msg_file,
+                      # "file_share": self._msg_file,
                       "me_message": self._msg_me}
-        self.symbols = {'plain': {'join': '->',
+
-                                  'leave': '<-',
-                                  'me': '*',
-                                  'file': '-',
-                                  'topic': '+',
-                                  'separator': '|'},
-                        'unicode': {'join': '⮊',
-                                    'leave': '⮈',
-                                    'me': '🟊',
-                                    'file': '📂',
-                                    'topic': '🟅',
-                                    'separator': '│'}}
         self.emoji = emoji.EMOJI.get(args.theme, {})
 
         self.channels = self._get_channels(args.channels)
         self.users = self.q(o.User).all()
-        self._slackid_pat = [re.compile(r'^(?P<replace>'
-                                        r'<@(?P<slackid>U[A-Z,0-9]+)\|.+>)'),
-                             re.compile('^(?P<replace>'
-                                        '<@(?P<slackid>U[A-Z,0-9]+)>)'),
-                             re.compile(r'.*(?P<replace>'
-                                        r'<@(?P<slackid>U[A-Z,0-9]+)\|.+>)'),
-                             re.compile('.*(?P<replace><@(?P<slackid>'
-                                        'U[A-Z,0-9]+)>)')]
 
     def generate(self):
-        """Generate raport it's a dummmy one - for use with none reporter"""
+        """Generate raport for each channel"""
-        return
+        for channel in self.channels:
+            messages = []
+            log_path = self.get_log_path(channel.name)
+            try:
+                os.unlink(log_path)
+            except IOError as err:
+                if err.errno != errno.ENOENT:
+                    raise
+            for message in self.q(o.Message).\
+                    filter(o.Message.channel == channel).\
+                    order_by(o.Message.ts).all():
+                messages.append(message)
+            self.write_msg(messages, log_path, channel)
 
     def get_log_path(self, name):
         """Return relative log file name """
         return os.path.join(self.out, name + self.ext)
 
-    def write_msg(self, message, log):
+    def write_msg(self, messages, log, channel):
         """Write message to file"""
-        raise NotImplementedError()
+        with open(log, "a", encoding='utf8') as fobj:
+            for message in messages:
+                data = self._process_message(message)
+                fobj.write(data['tpl'].format(**data))
+                if message.files:
+                    for _file in message.files:
+                        data = self._msg_file(message, _file)
+                        fobj.write(data['tpl'].format(**data))
+                # else:
+                    # data = self._process_message(message)
+                    # fobj.write(data['tpl'].format(**data))
 
     def _get_symbol(self, item):
         """Return appropriate item depending on the selected theme"""
@@ -87,46 +104,92 @@ class Reporter(object):
         for channel in all_channels:
             if channel.name in selected_channels:
                 result.append(channel)
+        return result
 
-    def _msg_join(self, msg, text):
+    def _process_message(self, msg):
-        """return formatter for join"""
+        """
-        return
+        Make changes to the text (replace slack ids, replace representation of
+        urls, substitute images etc) and return dict with data suitable to
+        display.
+        """
+        processor = self.types.get(msg.type, self._msg)
+        data = processor(msg)
+        data.update({'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+                     'tpl': "{date} {nick} {msg}"})
 
-    def _msg_leave(self, msg, text):
+        for emoticon in self.emoji:
-        """return formatter for leave"""
+            data['msg'] = data['msg'].replace(emoticon, self.emoji[emoticon])
-        return
 
-    def _msg_topic(self, msg, text):
+        return data
-        """return formatter for set topic"""
-        return
 
-    def _msg_me(self, msg, text):
+    def _msg_join(self, msg):
-        """return formatter for /me"""
+        """return data for join"""
-        return
+        return {'msg': msg.text,
+                'nick': self._get_symbol('join')}
 
-    def _msg_file(self, msg, text):
+    def _msg_leave(self, msg):
-        """return formatter for /me"""
+        """return data for leave"""
-        return
+        return {'msg': msg.text,
+                'nick': self._get_symbol('leave')}
+
+    def _msg_topic(self, msg):
+        """return data for set topic"""
+        return {'msg': msg.text,
+                'nick': self._get_symbol('topic')}
+
+    def _msg_me(self, msg):
+        """return data for /me"""
+        return {'msg': msg.user.name + ' ' + msg.text,
+                'nick': self._get_symbol('me')}
+
+    def _msg_file(self, msg, _file):
+        """return data for file"""
+        return {'msg': msg.text,
+                'nick': self._get_symbol('file')}
+
+    def _msg(self, msg):
+        """return data for all other message types"""
+        return {'msg': msg.text,
+                'nick': msg.user.name}
 
     def _filter_slackid(self, text):
         """filter out all of the id from slack"""
+        match = True
+        while match:
+            match = self.slackid_pat.search(text)
+            if not match:
+                return text
+
+            match = match.groupdict()
+            user = self.q(o.User).filter(o.User.slackid ==
+                                         match['slackid']).one()
+            text = text.replace(match['replace'], user.name)
+
+        return text
+
+
+class NoneReporter(Reporter):
+    """Dummy reporter used for fallback"""
+
+    def generate(self):
+        """Generate raport it's a dummmy one - for use with none reporter"""
         return
 
 
 class TextReporter(Reporter):
     """Text aka IRC reporter"""
     ext = '.log'
+    tpl = '{date} {nick:>{max_len}} {separator} {msg}\n'
 
     def __init__(self, args, query):
         super(TextReporter, self).__init__(args, query)
         utils.makedirs(self.out)
         self._max_len = 0
 
-        return
-
     def generate(self):
         """Generate raport"""
         for channel in self.channels:
+            messages = []
             log_path = self.get_log_path(channel.name)
             self._set_max_len(channel)
             try:
@@ -137,12 +200,9 @@ class TextReporter(Reporter):
             for message in self.q(o.Message).\
                     filter(o.Message.channel == channel).\
                     order_by(o.Message.ts).all():
-                self.write_msg(message, log_path)
+                messages.append(message)
 
-    def write_msg(self, message, log):
+            self.write_msg(messages, log_path, channel)
-        """Write message to file"""
-        with open(log, "a") as fobj:
-            fobj.write(self._format_message(message))
 
     def _set_max_len(self, channel):
         """calculate max_len for sepcified channel"""
@@ -154,133 +214,60 @@ class TextReporter(Reporter):
             if len(user_name) > self._max_len:
                 self._max_len = len(user_name)
 
-    def _format_message(self, msg):
+    def _process_message(self, msg):
         """
         Check what kind of message we are dealing with and do appropriate
         formatting
         """
-        msg_txt = self._filter_slackid(msg.text)
+        data = super(TextReporter, self)._process_message(msg)
-        msg_txt = self._fix_newlines(msg_txt)
+        data['msg'] = self._filter_slackid(data['msg'])
-        for emoticon in self.emoji:
+        data['msg'] = self._fix_newlines(data['msg'])
-            msg_txt = msg_txt.replace(emoticon, self.emoji[emoticon])
+        data['msg'] = self._remove_entities(data['msg'])
-        formatter = self.types.get(msg.type, self._msg)
+        data.update({'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+                     'max_len': self._max_len,
+                     'separator': self._get_symbol('separator'),
+                     'tpl': self.tpl})
+        return data
 
-        return formatter(msg, msg_txt)
+    def _msg_file(self, message, _file):
-
+        """return data for file"""
-    def _msg_join(self, msg, text):
+        if _file.filepath:
-        """return formatter for join"""
+            fpath = os.path.abspath(_file.filepath)
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+            fpath = pathlib.PurePath(fpath).as_uri()
-                'msg': text,
-                'max_len': self._max_len,
-                'separator': self._get_symbol('separator'),
-                'nick': self._get_symbol('join')}
-        return '{date} {nick:>{max_len}} {separator} {msg}\n'.format(**data)
-
-    def _msg_leave(self, msg, text):
-        """return formatter for leave"""
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
-                'msg': text,
-                'max_len': self._max_len,
-                'separator': self._get_symbol('separator'),
-                'nick': self._get_symbol('leave')}
-        return '{date} {nick:>{max_len}} {separator} {msg}\n'.format(**data)
-
-    def _msg_topic(self, msg, text):
-        """return formatter for set topic"""
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
-                'msg': text,
-                'max_len': self._max_len,
-                'separator': self._get_symbol('separator'),
-                'char': self._get_symbol('topic')}
-        return '{date} {char:>{max_len}} {separator} {msg}\n'.format(**data)
-
-    def _msg_me(self, msg, text):
-        """return formatter for /me"""
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
-                'msg': text,
-                'max_len': self._max_len,
-                'nick': msg.user.name,
-                'separator': self._get_symbol('separator'),
-                'char': self._get_symbol('me')}
-        return '{date} {char:>{max_len}} {separator} {nick} {msg}\n'.\
-            format(**data)
-
-    def _msg_file(self, msg, text):
-        """return formatter for file"""
-        groups = self._slackid_pat[0].match(msg.text).groupdict()
-        text = msg.text.replace(groups['replace'], '')
-        text = self._filter_slackid(msg.text)
-        filename = msg.file.filepath
-        if filename:
-            filename = os.path.relpath(msg.file.filepath, start=self.out)
         else:
-            filename = msg.file.url
+            fpath = 'does_not_exists'
 
-        if not filename:
+        return {'msg': _file.title + ' ' + fpath,
-            logging.warning("There is have a file object, but nothing has "
+                'nick': self._get_symbol('file'),
-                            "found. Name of the file object is `%s'",
+                'date': message.datetime().strftime("%Y-%m-%d %H:%M:%S"),
-                            msg.file.name)
-            filename = msg.file.name
-
-        text = self._filter_slackid(text)
-        text = self._remove_entities(text)
-        text = self._fix_newlines(text)
-
-        for emoticon in self.emoji:
-            text = text.replace(emoticon, self.emoji[emoticon])
-
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
-                'msg': text,
                 'max_len': self._max_len,
                 'separator': self._get_symbol('separator'),
-                'filename': filename,
+                'tpl': self.tpl}
-                'nick': msg.user.name,
-                'char': self._get_symbol('file')}
-        return ('{date} {char:>{max_len}} {separator} {nick} '
-                'shared file "{filename}"{msg}\n'.format(**data))
 
-    def _msg(self, msg, text):
+    def _msg(self, msg):
-        """return formatter for all other message types"""
+        """return data for all other message types"""
 
-        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+        data = super(TextReporter, self)._msg(msg)
-                'msg': text,
+        result = ''
-                'max_len': self._max_len,
-                'separator': self._get_symbol('separator'),
-                'nick': msg.user.name}
-        result = '{date} {nick:>{max_len}} {separator} {msg}\n'.format(**data)
 
         if msg.attachments:
             for att in msg.attachments:
                 if att.title:
-                    att_text = "\n" + att.title + '\n'
+                    att_text = att.title + '\n'
                 else:
-                    att_text = "\n" + self._fix_newlines(att.fallback) + '\n'
+                    att_text = self._fix_newlines(att.fallback) + '\n'
 
                 if att.text:
                     att_text += att.text
 
-                att_text = self._fix_newlines(att_text)
-                # remove first newline
-                att_text = att_text[1:]
-
                 result += att_text + '\n'
 
-        return result
+        data['msg'] += result.strip()
+        return data
 
     def _remove_entities(self, text):
         """replace html entites into appropriate chars"""
-        text = HTMLParser().unescape(text)
+        return html.parser.HTMLParser().unescape(text)
-
-    def _filter_slackid(self, text):
-        """filter out all of the id from slack"""
-        for pat in self._slackid_pat:
-            while pat.search(text):
-                groups = pat.search(text).groupdict('slackid')
-                user = [u for u in self.users
-                        if u.slackid == groups['slackid']][0]
-                text = text.replace(groups['replace'], user.name)
-
-        return text
 
     def _fix_newlines(self, text):
         """Shift text with new lines to the right with separator"""
@@ -292,11 +279,206 @@ class TextReporter(Reporter):
                             self._get_symbol('separator') + ' ')
 
 
+class StaticHtmlReporter(Reporter):
+    """Text-like, but with browsable, clickable links"""
+    ext = '.html'
+    index_templ = """<!DOCTYPE html>
+        <html>
+        <head>
+        <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
+        <title>%(title)s</title>
+        </head>
+        <body>
+        <div id="container">
+        %(msgs)s
+        </div>
+        </body>
+        </html>
+    """
+    index_list = """
+        <ul>
+            %s
+        </ul>
+    """
+    msg_head = """<!DOCTYPE html>
+        <html>
+        <head>
+        <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
+        <title>%(title)s</title>
+        """
+    msg_style = """
+        <style>
+        * {
+            font-family: sans-serif;
+        }
+        .log {
+            width: 100%;
+        }
+        .log tr:nth-child(even) {
+            background-color: #efefef;
+        }
+        .nick {
+            text-align: right;
+            white-space: nowrap;
+        }
+        .date {
+            white-space: nowrap;
+        }
+        td {
+            padding: 2px;
+        }
+        img {
+            max-height: 300px;
+            max-widht: 500px;
+        }
+        </style>
+        </head>
+        <body>
+        <div id="container">
+        <table class="log">
+    """
+    msg_foot = """
+        </table>
+        </div>
+        </body>
+        </html>
+    """
+    msg_line = """
+            <tr>
+                <td class="date">{date}</td>
+                <td class="nick">{nick}</td>
+                <td>{msg}</td>
+            </tr>
+        """
+
+    def __init__(self, args, query):
+        super(StaticHtmlReporter, self).__init__(args, query)
+        utils.makedirs(self.out)
+        self._max_len = 0
+
+    def generate(self):
+        """Generate raport"""
+        super(StaticHtmlReporter, self).generate()
+
+        with open(os.path.join(self.out, "index.html"), "w",
+                  encoding='utf8') as fobj:
+            content = {'title': 'index',
+                       'msgs': self.index_list % self._get_index_list()}
+            fobj.write(self.index_templ % content)
+
+    def write_msg(self, messages, log, channel):
+        """Write message to file"""
+        with open(log, "w", encoding='utf8') as fobj:
+            title = channel.name
+            if channel.topic:
+                title = channel.name + ' ' + channel.topic.value
+            fobj.write(self.msg_head % {'title': title})
+            fobj.write(self.msg_style)
+
+        super(StaticHtmlReporter, self).write_msg(messages, log, channel)
+
+        with open(log, "a", encoding='utf8') as fobj:
+            fobj.write(self.msg_foot)
+
+    def _get_index_list(self):
+        _list = []
+        for channel in sorted([c.name for c in self.channels]):
+            _list.append('<li><a href="%s">%s</a></li>' % (channel + '.html',
+                                                           channel))
+        return '\n'.join(_list)
+
+    def _process_message(self, msg):
+        """
+        Check what kind of message we are dealing with and do appropriate
+        formatting
+        """
+        data = super(StaticHtmlReporter, self)._process_message(msg)
+        data['msg'] = self._filter_slackid(data['msg'])
+        data.update({'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+                     'tpl': self.msg_line})
+        return data
+
+    def _msg_file(self, msg, _file):
+        """return data for file"""
+        if _file.filepath:
+            fpath = os.path.abspath(_file.filepath)
+            fpath = pathlib.PurePath(fpath).as_uri()
+        else:
+            fpath = 'does_not_exists'
+
+        _, ext = os.path.splitext(fpath)
+        if ext.lower() in ('.png', '.jpg', '.jpeg', '.gif'):
+            url = ('<img src="' + fpath + '" alt="' +
+                   _file.title + '">')
+        else:
+            url = ('<a href="' + fpath + '">' + _file.title + '</a>')
+
+        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+                'msg': self._filter_slackid(url + _file.title),
+                'tpl': self.msg_line,
+                'nick': self._get_symbol('file')}
+
+        for emoticon in self.emoji:
+            data['msg'] = data['msg'].replace(emoticon, self.emoji[emoticon])
+
+        return data
+
+    def _msg(self, msg):
+        """return processor for all other message types"""
+
+        match = self.url2_pat.match(msg.text)
+        text = msg.text
+        if match:
+            text = ''
+            for part in self.url2_pat.split(msg.text):
+                if 'http' in part:
+                    text += '<a href="' + part + '">' + part + '</a>'
+                else:
+                    text += part
+
+        data = {'date': msg.datetime().strftime("%Y-%m-%d %H:%M:%S"),
+                'msg': text,
+                'nick': msg.user.name}
+
+        link = '<a href="{url}">{title}</a>'
+        attachment_msg = []
+
+        if msg.attachments:
+            for att in msg.attachments:
+                if 'http' in att.fallback:
+                    match = self.url_pat.search(att.fallback)
+                    if not match:
+                        match = self.literal_url_pat.search(att.fallback)
+                    match = match.groupdict()
+
+                    if 'title' not in match:
+                        match['title'] = match['url']
+                        if att.title:
+                            match['title'] = att.title
+
+                    att_text = att.fallback.replace(match['replace'],
+                                                    link.format(**match))
+                else:
+                    match = self.url_pat.search(msg.text)
+                    if match:
+                        match = match.groupdict()
+                        match['title'] = att.fallback
+                        att_text = msg.text.replace(match['replace'],
+                                                    link.format(**match))
+                    else:
+                        att_text = att.fallback
+                attachment_msg.append(att_text)
+
+        data['msg'] += '<br>'.join(attachment_msg)
+        return data
+
+
 def get_reporter(args, query):
     """Return object of right reporter class"""
-    reporters = {'text': TextReporter}
+    reporters = {'text': TextReporter,
+                 'html': StaticHtmlReporter}
 
-    klass = reporters.get(args.format, Reporter)
+    klass = reporters.get(args.format, NoneReporter)
     if klass.__name__ == 'Reporter':
         logging.warning('None, or wrong (%s) formatter selected, falling to'
                         ' None Reporter', args.format)

slack_backup/utils.py

@@ -1,9 +1,12 @@
 """
 Some utils functions. Jsut to not copypaste the code around
 """
+from datetime import datetime
 import errno
 import os
 import logging
+import tempfile
+import hashlib
 
 
 def makedirs(path):
@@ -19,3 +22,36 @@ def makedirs(path):
                 logging.error("Cannot create `%s'. There is some file on the "
                               "way; cannot proceed.", path)
                 raise
+
+
+def get_temp_name(suffix='', prefix='tmp', dir=None, unlink=False):
+    """Return temporary file name"""
+    fdesc, fname = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir)
+    os.close(fdesc)
+    if unlink:
+        os.unlink(fname)
+    return fname
+
+
+def same_files(file1, file2):
+    """
+    Compare files by calculating hash for each of them. Return True if hash is
+    identical, False otherwise
+    """
+    with open(file1, 'rb') as fobj:
+        hash1 = hashlib.sha256(fobj.read())
+
+    with open(file2, 'rb') as fobj:
+        hash2 = hashlib.sha256(fobj.read())
+
+    return hash1.hexdigest() == hash2.hexdigest()
+
+
+def fromtimestamp(timestamp):
+    """
+    Return datetime object from provided timestamp. If timestamp argument is
+    falsy, datetime object placed in January 1970 will be retuned.
+    """
+    if not timestamp:
+        return datetime.utcfromtimestamp(0)
+    return datetime.fromtimestamp(timestamp)

tests/test_client.py

@@ -1,8 +1,6 @@
-from unittest import TestCase
+import copy
-try:
+import unittest
-    from unittest.mock import MagicMock
+from unittest import mock
-except ImportError:
-    from mock import MagicMock
 
 from slack_backup import client
 from slack_backup import objects as o
@@ -260,66 +258,67 @@ MSGS = {'messages': [{"type": "message",
                      {"display_as_bot": False,
                       "subtype": "file_share",
                       "username": "<@UCCCCCCCC|name2>",
-                      "file": {"thumb_960": "https://files.slack.com/files-tmb"
+                      "files": [{"thumb_960": "https://files.slack.com/files-"
-                                            "/hash/screenshot_960.png",
+                                              "tmb/hash/screenshot_960.png",
-                               "user": "UCCCCCCCC",
+                                 "user": "UCCCCCCCC",
-                               "size": 77222,
+                                 "size": 77222,
-                               "thumb_1024_h": 754,
+                                 "thumb_1024_h": 754,
-                               "timestamp": 1479407569,
+                                 "timestamp": 1479407569,
-                               "url_private_download": "https://files.slack.co"
+                                 "url_private_download": "https://files.slack."
-                                                       "m/files-pri/hsh/downlo"
+                                                         "com/files-pri/hsh/do"
-                                                       "ad/screenshot.png",
+                                                         "wnload/screenshot.pn"
-                               "thumb_360": "https://files.slack.com/files-tmb"
+                                                         "g",
-                                            "/hash/screenshot_360.png",
+                                 "thumb_360": "https://files.slack.com/files-"
-                               "username": "",
+                                              "tmb/hash/screenshot_360.png",
-                               "external_type": "",
+                                 "username": "",
-                               "thumb_64": "https://files.slack.com/files-tmb/"
+                                 "external_type": "",
-                                           "hash/screenshot_64.png",
+                                 "thumb_64": "https://files.slack.com/files-"
-                               "created": 1479407569,
+                                             "tmb/hash/screenshot_64.png",
-                               "ims": [],
+                                 "created": 1479407569,
-                               "groups": [],
+                                 "ims": [],
-                               "filetype": "png",
+                                 "groups": [],
-                               "thumb_1024": "https://files.slack.com/files-"
+                                 "filetype": "png",
-                                             "tmb/hash/screenshot_1024.png",
+                                 "thumb_1024": "https://files.slack.com/files-"
-                               "original_w": 1193,
+                                               "tmb/hash/screenshot_1024.png",
-                               "name": "Screenshot.png",
+                                 "original_w": 1193,
-                               "thumb_360_h": 265,
+                                 "name": "Screenshot.png",
-                               "is_public": True,
+                                 "thumb_360_h": 265,
-                               "thumb_960_h": 707,
+                                 "is_public": True,
-                               "original_h": 878,
+                                 "thumb_960_h": 707,
-                               "mimetype": "image/png",
+                                 "original_h": 878,
-                               "id": "F3405RRB5",
+                                 "mimetype": "image/png",
-                               "pretty_type": "PNG",
+                                 "id": "F3405RRB5",
-                               "editable": False,
+                                 "pretty_type": "PNG",
-                               "thumb_960_w": 960,
+                                 "editable": False,
-                               "thumb_80": "https://files.slack.com/files-tmb/"
+                                 "thumb_960_w": 960,
-                                           "hash/screenshot_80.png",
+                                 "thumb_80": "https://files.slack.com/files-"
-                               "comments_count": 0,
+                                             "tmb/hash/screenshot_80.png",
-                               "image_exif_rotation": 1,
+                                 "comments_count": 0,
-                               "thumb_160": "https://files.slack.com/files-tmb"
+                                 "image_exif_rotation": 1,
-                                            "/hash/screenshot_160.png",
+                                 "thumb_160": "https://files.slack.com/files-"
-                               "thumb_480_w": 480,
+                                              "tmb/hash/screenshot_160.png",
-                               "is_external": False,
+                                 "thumb_480_w": 480,
-                               "display_as_bot": False,
+                                 "is_external": False,
-                               "thumb_720_h": 530,
+                                 "display_as_bot": False,
-                               "channels": ["C00000001"],
+                                 "thumb_720_h": 530,
-                               "title": "Screenshot.png",
+                                 "channels": ["C00000001"],
-                               "thumb_480": "https://files.slack.com/files-tmb"
+                                 "title": "Screenshot.png",
-                                            "/hash/screenshot_480.png",
+                                 "thumb_480": "https://files.slack.com/files-"
-                               "url_private": "https://files.slack.com/files-"
+                                              "tmb/hash/screenshot_480.png",
-                                              "pri/hsh/screenshot.png",
+                                 "url_private": "https://files.slack.com/files"
-                               "mode": "hosted",
+                                                "-pri/hsh/screenshot.png",
-                               "thumb_1024_w": 1024,
+                                 "mode": "hosted",
-                               "permalink": "https://esm64.slack.com/files/"
+                                 "thumb_1024_w": 1024,
-                                            "name2/F3405RRB5/screenshot.png",
+                                 "permalink": "https://esm64.slack.com/files/"
-                               "thumb_480_h": 353,
+                                              "name2/F3405RRB5/screenshot.png",
-                               "public_url_shared": False,
+                                 "thumb_480_h": 353,
-                               "thumb_720": "https://files.slack.com/files-tmb"
+                                 "public_url_shared": False,
-                                            "/hash/screenshot_720.png",
+                                 "thumb_720": "https://files.slack.com/files-"
-                               "thumb_360_w": 360,
+                                              "tmb/hash/screenshot_720.png",
-                               "permalink_public": "https://slack-files.com/"
+                                 "thumb_360_w": 360,
-                                                   "hsh-7dbb96b758",
+                                 "permalink_public": "https://slack-files.com/"
-                               "thumb_720_w": 720},
+                                                     "hsh-7dbb96b758",
+                                 "thumb_720_w": 720}],
                       "type": "message",
                       "user": "UCCCCCCCC",
                       "bot_id": None,
@@ -327,7 +326,10 @@ MSGS = {'messages': [{"type": "message",
                               "<https://esm64.slack.com/files/name2/F3405RRB5/"
                               "screenshot.png|Screenshot.png>",
                       "ts": "1478107371.000052",
-                      "upload": True}],
+                      "upload": True},
+                     {'type': 'something else',
+                      'ts': '1502003415232.000001',
+                      "wibblr": True}],
         "ok": True,
         "latest": "1479501075.000020",
         "has_more": True}
@@ -348,6 +350,234 @@ MSG3 = {"ok": True,
         "has_more": False,
         "is_limited": False}
 
+STARRED = {"ok": True,
+           "oldest": "1479505026.000003",
+           "messages": [],
+           "has_more": False,
+           "is_limited": False}
+
+SHARED = {"type": "message",
+          "subtype": "file_share",
+          "text": "<@UAAAAAAAA> shared a file: <https://bla."
+          "slack.com/files/name%20lastname/F7ARMB4JU/17|"
+          "some_spreadsheet>",
+          "files": [{
+                   "id": "F6ABMB0Ja",
+                   "created": 1479147929,
+                   "timestamp": 1479147929,
+                   "name": "att name",
+                   "title": "some_spreadsheet",
+                   "mimetype": "application/vnd.google-apps."
+                   "spreadsheet",
+                   "filetype": "gsheet",
+                   "pretty_type": "GDocs Spreadsheet",
+                   "user":  'UAAAAAAAA',
+                   "editable": False,
+                   "size": 666,
+                   "mode": "external",
+                   "is_external": True,
+                   "external_type": "gdrive",
+                   "is_public": True,
+                   "public_url_shared": False,
+                   "display_as_bot": False,
+                   "username": "",
+                   "url_private": "https://docs.google.com/"
+                   "spreadsheets/d/name%20lastname/"
+                   "edit?usp=drivesdk",
+                   # removed useless thumb_* definition
+                   "image_exif_rotation": 1,
+                   "original_w": 1024,
+                   "original_h": 1449,
+                   "permalink": "https://bla.slack.com/files/"
+                   "name%20lastname/F7ARMB4JU/17",
+                   "channels": ["C00000001"],
+                   "groups": [],
+                   "ims": [],
+                   "comments_count": 0,
+                   "has_rich_preview": True
+                  }],
+          "user": 'UAAAAAAAA',
+          "upload": False,
+          "display_as_bot": False,
+          "username": "name lastname",
+          "bot_id": None,
+          "ts": "1479147929.000043"}
+
+PINNED = {'attachments': [{'fallback': 'blah',
+                           'id': 1,
+                           'image_bytes': 5,
+                           'image_height': 2,
+                           'image_url': 'http://fake.com/i.png',
+                           'image_width': 1,
+                           'original_url': 'http://fake.com/fake',
+                           'service_icon': 'http://fake.com/favicon.ico',
+                           'service_name': 'fake service',
+                           'text': 'the text',
+                           'title': 'Fake service title',
+                           'title_link': 'http://fake.com/fake'}],
+          'item_type': 'C',
+          'subtype': 'pinned_item',
+          'text': '<@UAAAAAAAA> pinned a message to this channel.',
+          'ts': '1479147929.000043',
+          'type': 'message',
+          'user': 'UAAAAAAAA'}
+
+EXTERNAL_DATA = {"bot_id": None,
+                 "display_as_bot": False,
+                 "files": [{
+                          "channels": [
+                                       "xxx"
+                                      ],
+                          "id": "F7ARMB4JU",
+                          "created": 1506819447,
+                          "timestamp": 1506819447,
+                          "name": "17",
+                          "title": "PT Card Count 9/30/17",
+                          "mimetype": "application/"
+                                      "vnd.google-apps.spreadsheet",
+                          "filetype": "gsheet",
+                          "pretty_type": "GDocs Spreadsheet",
+                          "user": "xxx",
+                          "editable": False,
+                          "size": 37583,
+                          "mode": "external",
+                          "is_external": True,
+                          "external_type": "gdrive",
+                          "is_public": True,
+                          "public_url_shared": False,
+                          "display_as_bot": False,
+                          "username": "",
+                          "url_private": "https://docs.google.com/"
+                                         "spreadsheets/d/xxx/edit?"
+                                         "usp=drivesdk",
+                          "thumb_64": "https://files.slack.com/files-tmb/"
+                                      "xxx-F7ARMB4JU-xxx/17_64.png",
+                          "thumb_80": "https://files.slack.com/files-tmb/"
+                                      "xxx-F7ARMB4JU-xxx/17_80.png",
+                          "thumb_360": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_360.png",
+                          "thumb_360_w": 254,
+                          "thumb_360_h": 360,
+                          "thumb_480": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_480.png",
+                          "thumb_480_w": 339,
+                          "thumb_480_h": 480,
+                          "thumb_160": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_160.png",
+                          "thumb_720": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_720.png",
+                          "thumb_720_w": 509,
+                          "thumb_720_h": 720,
+                          "thumb_800": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_800.png",
+                          "thumb_800_w": 800,
+                          "thumb_800_h": 1132,
+                          "thumb_960": "https://files.slack.com/files-tmb"
+                                       "/xxx-F7ARMB4JU-xxx/17_960.png",
+                          "thumb_960_w": 678,
+                          "thumb_960_h": 960,
+                          "thumb_1024": "https://files.slack.com/files-tmb"
+                                        "/xxx-F7ARMB4JU-xxx/17_1024.png",
+                          "thumb_1024_w": 724,
+                          "thumb_1024_h": 1024,
+                          "image_exif_rotation": 1,
+                          "original_w": 1024,
+                          "original_h": 1449,
+                          "permalink": "https://xxx.slack.com/files/xxx/"
+                                       "F7ARMB4JU/17",
+                          "groups": [],
+                          "ims": [],
+                          "comments_count": 0,
+                          "has_rich_preview": True}],
+                 "user": "xxx",
+                 "upload": False,
+                 "username": "xxx"}
+
+INTERNAL_DATA = {"bot_id": None,
+                 "display_as_bot": False,
+                 "files": [{"channels": ["zzz"],
+                            "comments_count": 1,
+                            "created": 1524724681,
+                            "display_as_bot": False,
+                            "editable": False,
+                            "external_type": "",
+                            "filetype": "jpg",
+                            "groups": [],
+                            "id": "id",
+                            "image_exif_rotation": 1,
+                            "ims": [],
+                            "initial_comment": {"comment": "bla",
+                                                "created": 1524724681,
+                                                "id": "yyy",
+                                                "is_intro": True,
+                                                "timestamp": 1524724681,
+                                                "user": "UAAAAAAAA"},
+                            "is_external": False,
+                            "is_public": True,
+                            "mimetype": "image/jpeg",
+                            "mode": "hosted",
+                            "name": "img.jpg",
+                            "original_h": 768,
+                            "original_w": 1080,
+                            "permalink": "https://fake.slack.com/files/"
+                                         "UAAAAAAAA/id/img.jpg",
+                            "permalink_public": "https://slack-files.com/"
+                                                "TXXXXXX-id-3de9b969d2",
+                            "pretty_type": "JPEG",
+                            "public_url_shared": False,
+                            "reactions": [{"count": 1,
+                                           "name": "thinking_face",
+                                           "users": ["U6P9KEALW"]}],
+                            "size": 491335,
+                            "thumb_1024": "https://files.slack.com/files-tmb/"
+                                          "TXXXXXX-id-123/img_1024.jpg",
+                            "thumb_1024_h": 728,
+                            "thumb_1024_w": 1024,
+                            "thumb_160": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_160.jpg",
+                            "thumb_360": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_360.jpg",
+                            "thumb_360_h": 256,
+                            "thumb_360_w": 360,
+                            "thumb_480": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_480.jpg",
+                            "thumb_480_h": 341,
+                            "thumb_480_w": 480,
+                            "thumb_64": "https://files.slack.com/files-tmb/"
+                                        "TXXXXXX-id-123/img_64.jpg",
+                            "thumb_720": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_720.jpg",
+                            "thumb_720_h": 512,
+                            "thumb_720_w": 720,
+                            "thumb_80": "https://files.slack.com/files-tmb/"
+                                        "TXXXXXX-id-123/img_80.jpg",
+                            "thumb_800": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_800.jpg",
+                            "thumb_800_h": 569,
+                            "thumb_800_w": 800,
+                            "thumb_960": "https://files.slack.com/files-tmb/"
+                                         "TXXXXXX-id-123/img_960.jpg",
+                            "thumb_960_h": 683,
+                            "thumb_960_w": 960,
+                            "timestamp": 1524724681,
+                            "title": "img.jpg",
+                            "url_private": "https://files.slack.com/files-pri/"
+                                           "TXXXXXX-id/img.jpg",
+                            "url_private_download": "https://files.slack.com/"
+                                                    "files-pri/TXXXXXX-id/"
+                                                    "download/img.jpg",
+                            "user": "UAAAAAAAA",
+                            "username": ""}],
+                 "subtype": "file_share",
+                 "text": "<@UAAAAAAAA> uploaded a file: <https://fake.slack."
+                         "com/files/UAAAAAAAA/id/img.jpg|img.jpg> and "
+                         "commented: bla",
+                 "ts": "1524724685.000201",
+                 "type": "message",
+                 "upload": True,
+                 "user": "UAAAAAAAA",
+                 "username": "bob"}
+
 
 class FakeArgs(object):
     token = 'token_string'
@@ -356,43 +586,44 @@ class FakeArgs(object):
     team = 'fake_team'
     database = None
     channels = None
+    url_file_to_attachment = False
 
     def __contains__(self, key):
         return hasattr(self, key)
 
 
-class TestApiCalls(TestCase):
+class TestApiCalls(unittest.TestCase):
 
     def test_channels_list(self):
         cl = client.Client(FakeArgs())
-        cl.slack.api_call = MagicMock(return_value=CHANNELS)
+        cl.slack.api_call = mock.MagicMock(return_value=CHANNELS)
         channels = cl._channels_list()
         self.assertListEqual(CHANNELS['channels'], channels)
 
     def test_users_list(self):
         cl = client.Client(FakeArgs())
-        cl.slack.api_call = MagicMock(return_value=USERS)
+        cl.slack.api_call = mock.MagicMock(return_value=USERS)
         users = cl._users_list()
         self.assertListEqual(USERS['members'], users)
 
     def test_channels_history(self):
         cl = client.Client(FakeArgs())
 
-        cl.slack.api_call = MagicMock(return_value=USERS)
+        cl.slack.api_call = mock.MagicMock(return_value=USERS)
-        cl.downloader._download = MagicMock(return_value=None)
+        cl.downloader._download = mock.MagicMock(return_value=None)
         cl.update_users()
 
-        cl.slack.api_call = MagicMock(return_value=CHANNELS)
+        cl.slack.api_call = mock.MagicMock(return_value=CHANNELS)
         cl.update_channels()
 
-        cl.slack.api_call = MagicMock()
+        cl.slack.api_call = mock.MagicMock()
         cl.slack.api_call.side_effect = [MSGS, MSG2, MSG3]
 
         channel = cl.q(o.Channel).filter(o.Channel.slackid ==
                                          "C00000001").one()
 
         msg, ts = cl._channels_history(channel, 0)
-        self.assertEqual(len(msg), 5)
+        self.assertEqual(len(msg), 6)
         self.assertEqual(ts, '1479501074.000032')
 
         msg, ts = cl._channels_history(channel, ts)
@@ -404,12 +635,12 @@ class TestApiCalls(TestCase):
         self.assertIsNone(ts)
 
 
-class TestClient(TestCase):
+class TestClient(unittest.TestCase):
 
     def test_update_users(self):
         cl = client.Client(FakeArgs())
-        cl.slack.api_call = MagicMock(return_value=USERS)
+        cl.slack.api_call = mock.MagicMock(return_value=USERS)
-        cl.downloader._download = MagicMock(return_value=None)
+        cl.downloader._download = mock.MagicMock(return_value=None)
         cl.update_users()
         users = cl.session.query(o.User).all()
         self.assertEqual(len(users), 4)
@@ -422,26 +653,29 @@ class TestClient(TestCase):
         self.assertEqual(users[0].slackid, 'UAAAAAAAA')
 
 
-class TestMessage(TestCase):
+class TestMessage(unittest.TestCase):
 
     def setUp(self):
         args = FakeArgs()
         args.channels = ['general']
 
         self.cl = client.Client(args)
-        self.cl.downloader.authorize = MagicMock()
+        self.cl.downloader.authorize = mock.MagicMock()
-        self.cl.slack.api_call = MagicMock(return_value=USERS)
+        self.cl.slack.api_call = mock.MagicMock(return_value=USERS)
-        self.cl.downloader._download = MagicMock(return_value=None)
+        self.cl.downloader._download = mock.MagicMock(return_value=None)
         self.cl.update_users()
 
-        self.cl.slack.api_call = MagicMock(return_value=CHANNELS)
+        self.cl.slack.api_call = mock.MagicMock(return_value=CHANNELS)
         self.cl.update_channels()
 
-        self.cl.slack.api_call = MagicMock()
+        self.cl.slack.api_call = mock.MagicMock()
 
-    def test_update_history(self):
+    @mock.patch('slack_backup.download.Download.download')
+    def test_update_history(self, download):
 
-        self.cl.downloader._download = MagicMock(return_value=None)
+        download.return_value = 'foo'
+
+        self.cl.downloader._download = mock.MagicMock(return_value=None)
         self.cl.slack.api_call.side_effect = [MSGS, MSG3]
         self.cl.update_history()
         self.assertEqual(len(self.cl.q(o.Message).all()), 5)
@@ -450,3 +684,157 @@ class TestMessage(TestCase):
         self.cl.update_history()
 
         self.assertEqual(len(self.cl.q(o.Message).all()), 6)
+
+
+class TestCreateMessage(unittest.TestCase):
+
+    @mock.patch('slack_backup.client.Client._file_data')
+    @mock.patch('slack_backup.client.Client._get_user')
+    def test_empty_message(self, gu, fd):
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value='aa')
+        cl.session = mock.MagicMock()
+        channel = o.Channel({'name': 'test', 'id': 'C00000001'})
+
+        cl._create_message({'type': 'message', 'text': ''}, channel)
+        cl.session.add.assert_not_called()
+
+    @mock.patch('slack_backup.client.Client._file_data')
+    @mock.patch('slack_backup.client.Client._get_user')
+    def test_message_with_reaction(self, gu, fd):
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value='aa')
+        cl.session = mock.MagicMock()
+        channel = o.Channel({'name': 'test', 'id': 'C00000001'})
+
+        cl._create_message(MSGS['messages'][1], channel)
+
+        msg = cl.session.add.call_args[0][0]
+        self.assertEqual(len(msg.attachments), 1)
+        self.assertEqual(len(msg.reactions), 1)
+        self.assertEqual(msg.reactions[0].name, '+1')
+        self.assertFalse(msg.is_starred)
+
+    @mock.patch('slack_backup.client.Client._get_user')
+    def test_starred_item(self, gu):
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value='aa')
+        cl.session = mock.MagicMock()
+        channel = o.Channel({'name': 'test', 'id': 'C00000001'})
+
+        data = {"type": "message",
+                "user": "UAAAAAAAA",
+                "text": "test",
+                "ts": "1479501074.000032",
+                "is_starred": True}
+        cl._create_message(data, channel)
+
+        msg = cl.session.add.call_args[0][0]
+        self.assertEqual(len(msg.attachments), 0)
+        self.assertEqual(msg.text, 'test')
+        self.assertEqual(msg.type, '')
+        self.assertTrue(msg.is_starred)
+
+    @mock.patch('slack_backup.client.Client._file_data')
+    @mock.patch('slack_backup.client.Client._get_user')
+    def test_external_file_upload(self, gu, fd):
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value='aa')
+        cl.session = mock.MagicMock()
+        channel = o.Channel({'name': 'test', 'id': 'C00000001'})
+
+        cl._create_message(SHARED, channel)
+
+        msg = cl.session.add.call_args[0][0]
+        self.assertEqual(len(msg.attachments), 0)
+        self.assertTrue('shared a file' in msg.text)
+        self.assertFalse(msg.is_starred)
+        self.assertEqual(msg.type, 'file_share')
+        fd.assert_called_once_with(msg, SHARED['files'][0])
+
+    @mock.patch('slack_backup.client.Client._file_data')
+    @mock.patch('slack_backup.client.Client._get_user')
+    def test_pinned_message_with_attachments(self, gu, fd):
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value='aa')
+        cl.session = mock.MagicMock()
+        cl._url_file_to_attachment = True
+        channel = o.Channel({'name': 'test', 'id': 'C00000001'})
+
+        cl._create_message(PINNED, channel)
+
+        msg = cl.session.add.call_args[0][0]
+        self.assertEqual(len(msg.attachments), 1)
+        self.assertEqual(msg.text, '<@UAAAAAAAA> pinned a message to this '
+                         'channel.')
+        self.assertEqual(msg.type, 'pinned_item')
+        self.assertEqual(msg.attachments[0].text, 'the text')
+        self.assertEqual(msg.attachments[0].title, 'Fake service title')
+
+
+class TestFileShare(unittest.TestCase):
+
+    @mock.patch('slack_backup.download.Download.download')
+    @mock.patch('slack_backup.utils.makedirs')
+    def test_file_data(self, md, dl):
+        dl.side_effect = ['some_path']
+
+        url = INTERNAL_DATA['files'][0]['url_private_download']
+
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value=url)
+        cl.session = mock.MagicMock()
+
+        msg = o.Message()
+        cl._file_data(msg, INTERNAL_DATA['files'][0])
+
+        self.assertIsNotNone(msg.files)
+        self.assertEqual(msg.files[0].filepath, 'some_path')
+
+    @mock.patch('slack_backup.download.Download.download')
+    @mock.patch('slack_backup.utils.makedirs')
+    def test_starred_file_data(self, md, dl):
+        dl.side_effect = ['some_path']
+
+        data = copy.deepcopy(INTERNAL_DATA)
+        data['files'][0]['is_starred'] = True
+        url = data['files'][0]['url_private_download']
+
+        cl = client.Client(FakeArgs())
+        cl.downloader._download = mock.MagicMock(return_value=url)
+        cl.session = mock.MagicMock()
+
+        msg = o.Message()
+        cl._file_data(msg, data['files'][0])
+
+        self.assertTrue(msg.is_starred)
+
+    @mock.patch('uuid.uuid4')
+    @mock.patch('slack_backup.download.Download.download')
+    @mock.patch('slack_backup.utils.makedirs')
+    def test_external_file_data(self, md, dl, uuid):
+        uuid.side_effect = ['aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee']
+        dl.side_effect = ['some_path']
+
+        cl = client.Client(FakeArgs())
+        cl.session = mock.MagicMock()
+
+        # pretend, that we are authorized
+        cl.downloader._authorized = True
+
+        msg = o.Message()
+        expexted_line = ('https://docs.google.com/spreadsheets/d/xxx/edit?'
+                         'usp=drivesdk --> '
+                         'assets/files/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee\n')
+
+        cl._file_data(msg, EXTERNAL_DATA['files'][0])
+        file_ = cl.session.add.call_args[0][0]
+        self.assertEqual(cl._dldata, [expexted_line])
+        self.assertEqual(file_.filepath,
+                         'assets/files/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee')
+
+        dl.assert_not_called()
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/test_config.py

@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import argparse
+import tempfile
+import os
+import unittest
+
+from slack_backup import config
+
+
+CONF = """\
+[common]
+channels=["one","two", "three"]
+database=dbfname.sqlite
+quiet=1
+verbose=2
+
+[generate]
+output=logs
+format=text
+theme=plain
+
+[fetch]
+user=someuser@address.com
+password=secret
+team=myteam
+token=xxxx-1111111111-222222222222-333333333333-r4nd0ms7uff
+"""
+
+
+class TestConfig(unittest.TestCase):
+
+    def setUp(self):
+
+        fd, self.confname = tempfile.mkstemp()
+        os.close(fd)
+
+        with open(self.confname, 'w') as fobj:
+            fobj.write(CONF)
+
+    def tearDown(self):
+        os.unlink(self.confname)
+
+    def test_config(self):
+
+        self.assertTrue(os.path.exists(self.confname))
+        self.assertTrue(os.path.isfile(self.confname))
+
+        args = argparse.Namespace()
+        args.config = None
+        args.parser = 'fetch'
+        args.verbose = 2
+
+        conf = config.Config()
+        conf.update(args)
+        self.assertDictEqual(vars(args), {'config': None,
+                                          'parser': 'fetch',
+                                          'verbose': 2,
+                                          'quiet': 0,
+                                          'channels': [],
+                                          'database': None,
+                                          'user': None,
+                                          'password': None,
+                                          'team': None,
+                                          'token': None,
+                                          'url_file_to_attachment': False,
+                                          'raw_dir': None})
+
+        args = argparse.Namespace()
+        args.config = self.confname
+        args.parser = 'fetch'
+        args.verbose = 2
+        args.quiet = None
+        args.channels = None
+        args.database = None
+        args.user = None
+        args.password = None
+        args.team = None
+        args.token = None
+
+        conf = config.Config()
+        conf.update(args)
+
+        self.assertEqual(conf._options['verbose'], 2)
+        self.assertListEqual(conf._options['channels'],
+                             ['one', 'two', 'three'])
+        self.assertEqual(conf._options['database'], 'dbfname.sqlite')
+        self.assertEqual(conf._options['user'], 'someuser@address.com')
+
+        self.assertDictEqual(vars(args), {'config': self.confname,
+                                          'parser': 'fetch',
+                                          'verbose': 2,
+                                          'quiet': 0,
+                                          'channels': ['one', 'two', 'three'],
+                                          'database': 'dbfname.sqlite',
+                                          'user': 'someuser@address.com',
+                                          'password': 'secret',
+                                          'team': 'myteam',
+                                          'token': 'xxxx-1111111111-'
+                                                   '222222222222-333333333333-'
+                                                   'r4nd0ms7uff',
+                                          'url_file_to_attachment': False,
+                                          'raw_dir': None})
+
+        # override some conf options with commandline
+        args = argparse.Namespace()
+        args.config = self.confname
+        args.parser = 'fetch'
+        args.verbose = None
+        args.quiet = 2
+        args.channels = ['foo']
+        args.database = None
+        args.user = 'joe'
+        args.password = 'ultricies'
+        args.team = ''
+        args.token = 'the token'
+
+        conf = config.Config()
+        conf.update(args)
+
+        self.assertDictEqual(vars(args), {'config': self.confname,
+                                          'parser': 'fetch',
+                                          'verbose': 0,
+                                          'quiet': 2,
+                                          'channels': ['foo'],
+                                          'database': 'dbfname.sqlite',
+                                          'user': 'joe',
+                                          'password': 'ultricies',
+                                          'team': '',
+                                          'token': 'the token',
+                                          'url_file_to_attachment': False,
+                                          'raw_dir': None})

tests/test_reporter.py

@@ -1,12 +1,7 @@
-# -*- coding: utf-8 -*-
+import unittest
+from unittest import mock
 
-from unittest import TestCase
+from slack_backup import reporters
-try:
-    from unittest.mock import MagicMock
-except ImportError:
-    from mock import MagicMock
-
-from slack_backup import reporters as r
 
 
 class FakeUser(object):
@@ -15,37 +10,46 @@ class FakeUser(object):
         self.name = name
 
 
-class TestReporter(TestCase):
+class TestReporter(unittest.TestCase):
 
     def setUp(self):
 
-        users = [FakeUser('U111AAAAA', 'user1'),
+        args = mock.MagicMock()
-                 FakeUser('U111BBBBB', 'user2'),
-                 FakeUser('U111CCCCC', 'funky_username1'),
-                 FakeUser('U111DDDDD', 'user-name°')]
-
-        args = MagicMock()
         args.output = 'logs'
-        query1 = MagicMock()
-        query1.all = MagicMock(return_value=users)
-        query = MagicMock(return_value=query1)
 
-        self.reporter = r.TextReporter(args, query)
+        self.one = mock.MagicMock()
+        query2 = mock.MagicMock()
+        query2.one = self.one
+        query1 = mock.MagicMock()
+        query1.filter = mock.MagicMock(return_value=query2)
+        query = mock.MagicMock(return_value=query1)
 
-    def test_regexp(self):
+        self.reporter = reporters.TextReporter(args, query)
+
+    def test_regexp1(self):
+        self.one.return_value = FakeUser('U111AAAAA', 'user1')
         text = 'Cras vestibulum <@U111AAAAA|user1> erat ultrices neque.'
+
         self.assertEqual(self.reporter._filter_slackid(text),
                          'Cras vestibulum user1 erat ultrices neque.')
 
+    def test_regexp2(self):
+        self.one.side_effect = [FakeUser('U111AAAAA', 'user1'),
+                                FakeUser('U111AAAAA', 'user1')]
         text = ('Cras vestibulum <@U111AAAAA|user1> erat ultrices '
                 '<@U111AAAAA|user1> neque.')
         self.assertEqual(self.reporter._filter_slackid(text),
                          'Cras vestibulum user1 erat ultrices user1 neque.')
 
-        text = ('<@U111BBBBB|user2>Praesent vel enim sed eros luctus '
+    def test_regexp3(self):
+        self.one.side_effect = [FakeUser('U111BBBBB', 'user2'),
+                                FakeUser('U111DDDDD', 'user-name°'),
+                                FakeUser('U111CCCCC', 'funky_username1')]
+        text = ('<@U111BBBBB|user2> Praesent vel enim sed eros luctus '
                 'imperdiet.\nMauris neque ante, <@U111DDDDD> placerat at, '
-                'mollis vitae, faucibus quis, <@U111CCCCC>leo. Ut feugiat.')
+                'mollis vitae, faucibus quis, <@U111CCCCC> leo. Ut feugiat.')
-
+        self.assertEqual(self.reporter._filter_slackid(text),
-# Praesent vel enim sed eros luctus imperdiet.  Vivamus urna quam, congue
+                         'user2 Praesent vel enim sed eros luctus '
-# vulputate, convallis non, cursus cursus, risus. Quisque aliquet. Donec
+                         'imperdiet.\nMauris neque ante, user-name° placerat '
-# vulputate egestas elit. Morbi dictum, sem sit amet aliquam.
+                         'at, mollis vitae, faucibus quis, funky_username1 '
+                         'leo. Ut feugiat.')

tox.ini

@@ -1,5 +1,5 @@
 [tox]
-envlist = py27,py34,{py27,py34}-flake8
+envlist = py3,py3-flake8
 
 usedevelop = True
 
@@ -10,19 +10,7 @@ commands = py.test --cov=slack_backup --cov-report=term-missing
 deps = -r{toxinidir}/requirements.txt
        -r{toxinidir}/test-requirements.txt
 
-[testenv:py27]
+[testenv:py3-flake8]
-usedevelop={[testenv]usedevelop}
+basepython = python3
-setenv = {[testenv]setenv}
-commands = {[testenv]commands}
-deps = {[testenv]deps}
-        mock
-
-[testenv:py27-flake8]
-basepython = python2.7
-deps = flake8
-commands = flake8 {posargs}
-
-[testenv:py34-flake8]
-basepython = python3.4
 deps = flake8
 commands = flake8 {posargs}