Skip to content

[Weblate Database] switch from utf8 to utf8mb4 charset.

Originally created by @hefee on #17343 (Redmine)

We are facing utf-8 encoing issues with the mariadb database.

those issues point to
https://github.com/WeblateOrg/weblate/issues/2218
https://github.com/WeblateOrg/weblate/issues/1054

and those pointing to the documentation:

https://docs.weblate.org/en/weblate-3.5.1/admin/install.html#mysql-or-mariadb

the proposed solution is to switch to `utf8mb4`.

The database already using `utf8mb4` but not all tables. At least `trans_component` is using `utf8` instead of `utf8mb4`.

2019-12-12 15:11:40,251 - UWC - root(ERROR): Adding new component triggered by 'wiki/src/news/celebrating_10_years/mafe.de.po' failed:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/mysql/base.py", line 101, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 226, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorvalue
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 217, in execute
    res = self._query(query)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 378, in _query
    rowcount = self._do_query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
    db.query(q)
  File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8D\\xA0 (...' for column 'source' at row 1")