Trello Import Issues

Hi, like probably many others my team is trying to migrate from Trello to Vikunja (which is looking great btw!).

I’m trying to self-host Vikunja but I’m having an issue where the Trello import doesn’t seem to do anything.

My steps:
(I tried this with both https:/ /vikunja.mydomain.org and http:/ /localhost:3456)

Note: I replaced https:// with https:/ / as my new account isn’t allowed to post more than 2 links.

I created a docker-compose.yml file:

version: '3'

services:
    vikunja:
        image: vikunja/vikunja
        environment:
            VIKUNJA_SERVICE_PUBLICURL: https:/ /vikunja.mydomain.org
            VIKUNJA_DATABASE_HOST: db
            VIKUNJA_DATABASE_PASSWORD: changeme
            VIKUNJA_DATABASE_TYPE: mysql
            VIKUNJA_DATABASE_USER: vikunja
            VIKUNJA_DATABASE_DATABASE: vikunja
            VIKUNJA_SERVICE_JWTSECRET: <a super secure random secret>
        ports:
            - 3456:3456
        volumes:
            - ./files:/app/vikunja/files
            - ./config.yml:/etc/vikunja/config.yml
        depends_on:
            db:
                condition: service_healthy
        restart: unless-stopped
    db:
        image: mariadb:10
        command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
        environment:
            MYSQL_ROOT_PASSWORD: supersecret
            MYSQL_USER: vikunja
            MYSQL_PASSWORD: changeme
            MYSQL_DATABASE: vikunja
        volumes:
            - ./db:/var/lib/mysql
        restart: unless-stopped
        healthcheck:
            test: ["CMD-SHELL", "mysqladmin ping -h localhost -u $$MYSQL_USER --password=$$MYSQL_PASSWORD"]
            interval: 2s
            start_period: 30s

And a config.yml file to house the trello config (I mounted this to /etc/vikunja in the compose yml):

migration:
  trello:
    # Whether to enable the trello migrator or not
    enable: true
    # The client id, required for making requests to the trello api
    # You need to register your vikunja instance at https:/ /trello.com/app-key (log in before you visit that link) to get this
    key: mykey
    # The url where clients are redirected after they authorized Vikunja to access their trello cards.
    # This needs to match the url you entered when registering your Vikunja instance at trello.
    # This is usually the frontend url where the frontend then makes a request to /migration/trello/migrate
    # with the code obtained from the trello api.
    # Note that the vikunja frontend expects this to end on /migrate/trello.
    redirecturl: https:/ /vikunja.mydomain.org/migrate/trello

Then I executed the commands

mkdir $PWD/files
chown 1000 $PWD/files
sudo docker-compose up -d

I created a user, logged in and went to the import settings and selected Trello.
Then was redirected to the Trello page (where I’ve previously gotten my key from and added my https:/ /vikunja.mydomain.org to the whitelist.
I was redirected back to Vikunja which said

Vikunja will now import your lists/projects, tasks, notes, reminders and files from Trello. As this will take a while, we will send you an email once done. You can close this window now.

I’ve waited but didn’t see any changes, and because there are no other indications on the website that something is going on or failing I don’t know what to do.

If it helps, I saw that there were some api requests in the docker logs, like:
GET 200 /api/v1/migration/trello/auth
POST 200 /api/v1/migration/trello/migrate
GET 200 /api/v1/migration/trello/status

I also got an email from Trello that I have authorized Vikunja Migration to my account.

Is there any other info that I could check?

Some additional suggestions I had in case you’re interested:

  • If possible I think it would be very helpful for new people trying Vikunja if the documentation about migration would be expanded or made more obvious. At the start, I wasn’t even aware that there is a Trello import. There is https:/ /vikunja.io/docs/config-options/ and some posts here which I used to eventually figured out how to enable the Trello import, but a dedicated setup documentation page listing the steps necessary would be great to have (especially since I assume a lot of people will look into migrating from Trello in the near future). Also a link to the documentation on the /migrate page wouldn’t hurt imo.

  • The Trello import looks like it will import all Trello projects that you have. I think some sort of way to select what you want to import specifically would be great. Maybe even something where you don’t have to connect your Trello account but instead you can just import the JSON file that you can export from every Trello board. Though I’m also looking to import image attachments from Trello to Vikunja, which is probably not something a JSON importer could do without connecting the Trello account because the JSON export just has links to Trello-hosted files which are usually only accessible when you’re logged in.

There have been a few fixes and improvements in the Trello migration lately, it’s possible that it works for you when you use the last unstable build. Can you check that?

I’ll happily accept a PR to add this to the docs.

I haven’t looked into the Trello json export, my hunch would be that the json file contains URL which the importer could download. (Requesting an attachment from the api also returns a unique link to the file).

Hey, thanks for the response!
I’ve tried the unstable build but didn’t get any results yet (after waiting for more than an hour).

I should say that I have a very extensive selection of projects 50+ heavily filled projects.
Does the importer wait for everything to imported before it’s displaying anything in Vikunja, or does in work on a project by project basis?
In case it does it all at once, I’m suspecting that it either just takes very long for me, or that something in the process errored. Maybe Trello also rate limits me at some point.
Having the option to select one or multiple Trello boards to import would be great to troubleshoot this (be it a display to select detected projects to import or just uploading/pasting a json file).

I’ll happily accept a PR to add this to the docs.

I’ve prepared something in the meantime. Can you activate my account?

I haven’t looked into the Trello json export, my hunch would be that the json file contains URL which the importer could download. (Requesting an attachment from the api also returns a unique link to the file).

Yeah the json export has these kinds of links exactly. Would be nice to get that feature, I suppose the main work would be parsing the json structure. Not sure how much of the current importer could be reused for that.

Update: It finally worked!
It seems like it just took a very long time. I’m unsure if the unstable version is required for this right now, but I’m staying on that version for now at least.
Seeing that my workflow to get the import working was successful, I will create the PR once I’m able to :slight_smile:

One note about the import: I’m not sure if the Trello data tells what attachment is currently the cover of a task, because this info was lost in the import. This means we have to go through the tasks and set covers which is not too bad as all the data seems to be there. Just in case this feature can be added that would be even better!
Additionally, if it would be possible to detect the structure of workspaces in Trello by creating parent projects per workspace which the individual projects in each workspace would be added in, that would be very convenient.

In any case, thank you so much!

Actually, I correct that no data was lost. It seems like a considerable amount of images are missing from the import. It’s quite weird but the only pattern I see is that images from around before summer 2020 are missing, do you have any idea why that could be?

Sorry for the spam, looking at the source code and the json export, the problem seems to be that older attachments didn’t have the mime type set.
The importer currently requires a mime type for the import:

Maybe there can be some pre-defined mime types for some file extension?
I’d love to help here as well but as I have limited Go experience and no dev setup I’d rather have someone more experienced fix this.

Yes, it does it all at once. Even if the import crashes mid-way, it will roll back any incomplete data.

Done!

Maybe there are things which can be shared between the two, but it should be relatively straight forward. There are already things in place to make file-based migration easier.

Do you think the workspace projects should be top level projects or nested under a “Migrated from Trello” project?

The problem here is what the quote in that line of code in the trello import says: Attachments can be more than just files, in that case they may still contain a url but not of the downloadable kind. I haven’t looked into it yet but I guess there isn’t really a way to make that distiction when not using the mime type.

I’ve created a PR for updating the docs here :slight_smile: #2183 - Add migrations setup doc - vikunja - Gitea

Yeah I think they should still be nested under “Migrated from Trello” to make it obvious in case there is already an existing project with the same name. It takes a bit of time to reassign the parent projects but I think that’s the best solution. (Having drag and drop in the projects list would be quite useful for reassigning parent projects btw, but I don’t want to keep flooding you with feature requests :sweat_smile:)

Oh, I see. I hope that there is a way though!. I’ll try to look into this a bit myself but as I’ve said, limited go knowledge :frowning:.

Btw, one very helpful thing would be if the covers set directly. I think the library the Trello importer is using has a field that should tell the right cover, so that would hopefully be simple to do? trello/card.go at 905f0387428bc7d5e01ed57fe830f5a60744532b · adlio/trello · GitHub

1 Like

@kolaente It seems like this should do the trick.

mime.TypeByExtension(filepath.Ext(name))

This example code should work, I think:

var extension = filepath.Ext(attachment.URL)
// In case this is not pointing to a file an argument could be made to include the url in the card description so the info doesn't get lost?
if extension == "" {
    log.Debugf("[Trello Migration] Attachment %s does not have a mime type, and the url %s is not pointing to a file, not downloading", attachment.ID, attachment.URL)
    continue
}

attachment.MimeType = mime.TypeByExtension(extension)
if attachment.MimeType == "" {
    log.Debugf("[Trello Migration] Attachment %s does not have a mime type, extension %s couldn't be mapped to a mime type either, not downloading", attachment.ID, extension)
    continue
}

IIRC, the task cover image functionality was added after the Trello importer already worked and never backported.

Looks good. Do you know if the attachment url will always contain a file ending?

I see, that makes sense! Do you want me to look into this and write up another example code snippet?

No, there can also be attached links to something like google docs, like https://docs.google.com/spreadsheets/d/<id>/edit. From my testing it seems like in these cases the file extension will be an empty string (which is caught by the if branch).

Cover images are now implemented: ca0de680ad Please check with the next unstable build.

Can there be file attachments without an extension?

Thanks a lot! I will probably wait for the mime type fix to be implemented (because it will take another couple hours to import) but after that I’ll definitely try it.

From what I’ve seen all valid files have an extension. I don’t think that’s possible unless the file was uploaded with an invalid file extension but then it wouldn’t work in Trello either I suppose.

Looks like the attachments have an isUploaded flag. I’ve changed the implementation in 22dcedcd7d so that it only takes that into account, not the mime type. Please check with the next unstable build if all attachments are migrated correctly.

Oh, that is a good point, I see the flag too. I’ll try the build and report back with the results!

@kolaente It seems like the latest build failed, apparently due to a timeout

level=error msg="Running error: context loading failed: failed to load packages: timed out to load packages: context deadline exceeded"

level=error msg="Timeout exceeded: try increasing it by passing --timeout option"

Error: exit status 4

Can you restart it? Drone CI

Restarted it, seems to have worked now. Please check again.

Hey, it’s me again. I’ve tried the Trello importer on the latest unstable branch. When you wrote this on Sunday. No I have about 2-3 times more boards to import from Trello because I didn’t have access to everything that needs to be migrated before. After a day I didn’t see any new project being added and Postman would return for /api/v1/migration/trello/status that “finished_at” is still at the min date value indicating the migration isn’t done yet.

I did see some weird behavior in the previously existing boards (which were part of the first import that succeeded on the previous unstable branch). It seemed like some of the tasks were part of multiple Buckets in the Kanban view at the same time. When I scrolled at the bottom of a Bucket, it would load more tasks but ones from seemingly different Buckets. Not sure what happened there, maybe some bug in unstable or a side effect from the importer running the import again after it was completed before?
Anyway, because of this weird behavior I decided to restart the process with a fresh setup, so I deleted all the database files and so on and started fresh.

Now I’m waiting since about 48h hours.

{
    "id": 1,
    "migrator_name": "trello",
    "started_at": "2024-03-11T21:49:55Z",
    "finished_at": "0001-01-01T00:00:00Z"
}

Do you recommend that I wait some more time? I did increase the amount of boards needed to import by 2-3 times like I said before, but I would have guessed that it would be done by now? Previously the import took around 4 hours iirc.
Is there any way to see if the import is still working correctly or if there was any issue during the process? Thanks a lot for your help and sorry for this massive block of text :sweat_smile:

Well the longest import I’ve heard of was ~ 2h, maybe you don’t need to wait 48h.

There should be debug log messages during the import. If it fails with an error, that will be logged too.

I found this in the docker logs, roughly 2.5 hours after I started the import. It says it was successfully, but I don’t see any new projects.

{"log":"2024-03-11T23:46:57.383478996Z: INFO\u0009▶ migration/Migrate 05f\u001b[0m Ran all migrations successfully.\n","stream":"stdout","time":"2024-03-11T23:46:57.383800263Z"}

Is it possible when the Ran all migrations successfully message is displayed but finished_at isn’t set that there is some other error in the process?

I will restart the import and see if it works this time.