All data deleted when upgrading & can't restore

geco · 6 January 2024 14:03

1.0 The Issue

After restoring a database dump from vikunja/api:0.20 using vikunja dump and vikunja restore my data is accessible. However, upgrading the API image to: vikunja/api:0.21 or vikunja/api:latest all my data disappears. And I get the following error:

An error occurred:
Error: Error while refreshing user info:

Please check if the api url is correct.

1.1 What I have tried

Restore dump to vikunja/api:0.21.

"could not restore table data for table users: Error 1054 (42S22): Unknown column 'default_list_id' in 'field list'"

Restore dump to vikunja/api:latest

 CRITICAL        ▶ db/Restore 2ca Could not find table definition for table lists

Restore dump to vikunja/api:0.20

Success!

After restoring dump to v0.20 (step 4), update the container to v0.21

Data is deleted.

After restoring dump to v0.20 (step 4), update the container to latest

Data is deleted.

2.0 Setup

2.1 System details

Arch: x64
OS: Ubuntu-Server

2.2 Configuration

docker-compose.yml

version: '3'

services:
  vikunja_db:
    container_name: vikunja_db
    image: mariadb:10
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    environment:
      MYSQL_ROOT_PASSWORD: REDACTED
      MYSQL_USER: REDACTED
      MYSQL_PASSWORD: REDACTED
      MYSQL_DATABASE: vikunja
    volumes:
      - REDACTED/vikunja/db:/var/lib/mysql
    networks:
      - vikunja_db
    restart: unless-stopped
  vikunja_api:
    container_name: vikunja_api
    image: vikunja/api:0.20
    environment:
      VIKUNJA_DATABASE_HOST: vikunja_db
      VIKUNJA_DATABASE_PASSWORD: REDACTED
      VIKUNJA_DATABASE_TYPE: mysql
      VIKUNJA_DATABASE_USER: REDACTED
      VIKUNJA_DATABASE_DATABASE: REDACTED
      VIKUNJA_SERVICE_JWTSECRET: REDACTED
      VIKUNJA_SERVICE_FRONTENDURL: https://REDACTED.DOMAIN/
    volumes:
      - REDACTED/vikunja/data:/app/vikunja/files
    depends_on:
      - vikunja_db
    restart: unless-stopped
    networks:
      - reverse_proxy
      - vikunja_db

  vikunja_frontend:
    container_name: vikunja_frontend
    image: vikunja/frontend:0.20.5
    restart: unless-stopped
    environment:
      VIKUNJA_ALLOW_ICON_CHANGES: "false"
    networks:
      - no-internet
      - reverse_proxy

networks:
  vikunja_db:
    external: false
  reverse_proxy:
    external:
      name: reverse_proxy
  no-internet:
    external: true

kolaente · 6 January 2024 17:36

Restoring dumps with newer versions is not really supported. You should always restore the dump with the same version you’ve used to create the dump, then upgrade.

How did you verify your data is gone? Did you check the database?

geco · 6 January 2024 18:16

To verify the data is deleted I ran:

MariaDB [vikunja]> select email from users;
Empty set (0.000 sec)

MariaDB [vikunja]> select * from tasks;
Empty set (0.001 sec)

I’m not sure how to proceed if updating just erases everything.

Also, testing from a blank DB and new container on v0.20 (I then created some sample tasks), if I upgrade that container to v0.21 or :latest, everything disappears within the web UI. Although, I’m still able to login and the database contains the tasks.

kolaente · 6 January 2024 20:43

Is there anything in the logs during/after upgrading? Can you verify the data is present when you restore from dump on 0.20?

Vikunja won’t remove data on its own during upgrade, especially not erase everything.

:latest is 0.22 currently, not 0.21.

geco · 6 January 2024 21:08

Yes, it visually shows when logged in and within the DB.

Logs dump:

info: creating the new user vikunja with 1000:1000
usermod: no changes
2024-01-06T21:01:24.164339292Z: INFO	▶ config/InitConfig 001 No config file found, using default or config from environment variables.
2024-01-06T21:01:24.16592887Z: CRITICAL	▶ migration/Migrate 003 Migration failed: dial tcp 192.168.208.2:3306: connect: connection refused
info: creating the new user vikunja with 1000:1000
usermod: no changes
2024-01-06T21:01:24.875265478Z: INFO	▶ config/InitConfig 001 No config file found, using default or config from environment variables.
2024-01-06T21:01:25.529797667Z: INFO	▶ migration/Migrate 0bf Ran all migrations successfully.
2024-01-06T21:01:25.530034112Z: INFO	▶ models/RegisterReminderCron 0c0 Mailer is disabled, not sending reminders per mail
2024-01-06T21:01:25.530149555Z: INFO	▶ models/RegisterOverdueReminderCron 0c1 Mailer is disabled, not sending overdue per mail
2024-01-06T21:01:25.530429663Z: INFO	▶ cmd/func27 0c3 Vikunja version v0.22.0
⇨ http server started on [::]:3456
2024-01-06T21:01:37.956022134Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 4.163721ms - REDACTED
2024-01-06T21:01:37.962102945Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 1.631768ms - REDACTED
2024-01-06T21:01:37.968221711Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 2.515491ms - REDACTED
2024-01-06T21:01:37.969698841Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 3.837035ms - REDACTED
2024-01-06T21:01:48.077535118Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 2.768922ms - REDACTED
info: creating the new user vikunja with 1000:1000
usermod: no changes
2024-01-06T21:01:58.848271041Z: INFO	▶ config/InitConfig 001 No config file found, using default or config from environment variables.
2024-01-06T21:01:58.872187002Z: INFO	▶ migration/Migrate 05a Ran all migrations successfully.
2024-01-06T21:01:58.872292157Z: INFO	▶ models/RegisterReminderCron 05b Mailer is disabled, not sending reminders per mail
2024-01-06T21:01:58.872348294Z: INFO	▶ models/RegisterOverdueReminderCron 05c Mailer is disabled, not sending overdue per mail
2024-01-06T21:01:58.872637911Z: INFO	▶ cmd/func27 05e Vikunja version v0.22.0
⇨ http server started on [::]:3456
2024-01-06T21:02:08.243234589Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 2.469436ms - REDACTED
2024-01-06T21:02:18.258205786Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 3.780879ms - REDACTED
2024-01-06T21:02:19.511888181Z: WEB 	▶ REDACTED  GET 200 /api/v1/info 866.6µs - REDACTED
2024-01-06T21:02:19.572497191Z: WEB 	▶ REDACTED  GET 401 /api/v1/lists/2/tasks?sort_by[]=position&order_by[]=asc&filter_by[]=done&filter_value[]=false&filter_comparator[]=equals&filter_concat=and&page=1 159.71µs - REDACTED
2024-01-06T21:02:19.572918587Z: WEB 	▶ REDACTED  GET 401 /api/v1/lists/2 93.083µs - REDACTED
2024-01-06T21:02:19.574279473Z: WEB 	▶ REDACTED  GET 401 /api/v1/lists/2/tasks?sort_by[]=position&order_by[]=asc&filter_by[]=done&filter_value[]=false&filter_comparator[]=equals&filter_concat=and&page=1 134.6µs - REDACTED
2024-01-06T21:02:22.80006886Z: WEB 	▶ REDACTED  GET 200 /api/v1/info 362.057µs - REDACTED
2024-01-06T21:02:26.308040251Z: WEB 	▶ REDACTED  POST 200 /api/v1/login 200.797316ms - REDACTED
2024-01-06T21:02:26.320349934Z: WEB 	▶ REDACTED  GET 200 /api/v1/user 1.575764ms - REDACTED
2024-01-06T21:02:26.366855913Z: WEB 	▶ REDACTED  GET 404 /api/v1/namespaces?is_archived=true&page=1 193.112µs - REDACTED
2024-01-06T21:02:26.376473895Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 9.865049ms - REDACTED
2024-01-06T21:02:26.390381276Z: WEB 	▶ REDACTED  GET 200 /api/v1/labels?page=1 23.669161ms - REDACTED
2024-01-06T21:02:26.420301581Z: WEB 	▶ REDACTED  GET 404 /api/v1/lists/2 130.87µs - REDACTED
2024-01-06T21:02:26.420527028Z: WEB 	▶ REDACTED  GET 404 /api/v1/lists/2/tasks?sort_by[]=position&order_by[]=asc&filter_by[]=done&filter_value[]=false&filter_comparator[]=equals&filter_concat=and&page=1 319.963µs - REDACTED
2024-01-06T21:02:26.435357091Z: WEB 	▶ REDACTED  GET 404 /api/v1/lists/2/tasks?sort_by[]=position&order_by[]=asc&filter_by[]=done&filter_value[]=false&filter_comparator[]=equals&filter_concat=and&page=1 113.871µs - REDACTED
2024-01-06T21:02:26.499532334Z: WEB 	▶ REDACTED  GET 200 /api/v1/avatar/admin?size=50&=1704574945783 133.137438ms - REDACTED
2024-01-06T21:02:31.67997309Z: WEB 	▶ REDACTED  POST 200 /api/v1/user/token 1.962685ms - REDACTED
2024-01-06T21:02:31.692776929Z: WEB 	▶ REDACTED  GET 200 /api/v1/avatar/admin?size=50&=1704574951150 1.617878ms - REDACTED
2024-01-06T21:02:31.693392553Z: WEB 	▶ REDACTED  GET 200 /api/v1/user 2.015405ms - REDACTED
2024-01-06T21:02:31.703958564Z: WEB 	▶ REDACTED  GET 200 /api/v1/avatar/admin?size=50&=1704574951167 1.379422ms - REDACTED
2024-01-06T21:02:36.365906068Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 2.07636ms - REDACTED
2024-01-06T21:02:47.148901746Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 3.706048ms - REDACTED
2024-01-06T21:02:57.243782952Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 3.780412ms - REDACTED
2024-01-06T21:03:07.357268059Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 3.914461ms - REDACTED
2024-01-06T21:03:17.547765885Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 1.533253ms - REDACTED
2024-01-06T21:03:27.702543636Z: WEB 	▶ REDACTED  GET 200 /api/v1/notifications?page=1 2.215366ms - REDACTED

The data is deleted regardless if the database was recovered using restore or a completely new on on v0.20 was created and upgraded to v0.21 or v0.22.

When I login via the web UI nothing shows up and I get the error as a toast “Error: not found”, which I assume to be related to this request which fails:

GET https://REDACTED/api/v1/namespaces?is_archived=true&page=1

Status: 404

kolaente · 6 January 2024 22:27

Does it get deleted if you simply restart the stack?

You should always upgrade the frontend when you upgrade the api. The frontend tries to fetch all namespaces but namespaces were removed in 0.21.0.

geco · 6 January 2024 22:44

It persists after docker restart ... and docker compose down && docker compose up -d.

I overlooked this. After upgrading both the api and frontend from v0.20 to v0.21 to v0.22 it worked! I have no idea why keeping the fronted as v0.20 resulted in the db erasing if the Vikunja was restored from a the CLI.

Either way, thank you for your help.

kolaente · 6 January 2024 23:00

Yeah that still sounds strange. The frontend doesn’t even have the capability to erase data.

You might want to check out the backup guide for the future, now that everything works: What to backup | Vikunja