Offline capabilities of the front-end?

I apologies to make another post for yet another question ! I hope that won’t be too much.

I just wanted to inquire about the possibility of adding offline capabilities to the front-end of Vikunja; meaning that even while offline, changes could be made that could then be uploaded to the database when connection is restored.

Specifically, I’m thinking as to how Nextcloud Tasks is working right now, and how it can be synchronized with a phone app such as Tasks .org. Tasks .org works perfectly fine offline, and tasks can be edited, added, removed or ticked even without an internet connection. Tasks.org can then sync the changes with the tasks on Nextcloud, which are in ICal format, if I remember correctly.

This functionality is absolutely amazing to edit tasks on the go, or to stay organized even during vacations or field trips where internet isn’t always available. It seems that Vikunja doesn’t have it for now, I wondered it could be done someday. Since Vikunja works with a back-end, I guess that this will be more difficult to do.

Thank you in advance !

1 Like

No worries, that’s what the forum is made for :slightly_smiling_face:

I have thought about that at the beginning especially for the mobile app. The problem are conflicts: How do you handle situations where you are on one device A (offline), editing a task and on another device B (online) which edits the same task? When A comes online again it would override the changes made by B without any special handling implemented. Not sure how tasks.org handles this.
That being said offline-sync could be implemented, it is just not that easy so I don’t think it would happen anytime soon.

In the tasks.org setting your Nextcloud is the backend, that doesn’t change much.

The fact that this might be possible is super exciting !

I think that the problem that you’re describing concerns merging conflicts, maybe ? It seems to be the same as when trying to merge two branches on a git that have had different modification since their separation. I imagine that there are libraries for such situations.

I have no idea how tasks.org deals with it; but since tasks.org and Nextcloud Tasks both work with the ICal protocol, I imagine that their might be unified rules with this protocol. Vikunja could use the same rules. Duplicates could also be created when conflicts are detected (with a clear mention), or the user could be asked for input.

If you’re interested, I could run some tests with Tasks.org to see how things work.

Yup, that’s a merge conflict. I don’t think there’s a way to automagically™ fix those with a library, if that would be the case I’m pretty sure git would do it and we wouldn’t need to deal with merge conflicts. The most obvious (and the least destructive) way to fix those conflicts would be to ask the user for input about the changes they want to keep. That would need some checks to prevent asking in cases where only different fields of some entity have changed, but on different times offline and online.

If you find out anything about how ical solves this, I’d be happy to know.

Sorry if I’ve been a bit optimistic. I agree that it might be more complex than I thought, even if automagical™ fixes are always nice.

I’ve taken a look at some classes in the code of Tasks .org on github, and in particular the CaldavSynchronizer class.

I think that it’s javascript, which I’m not really good at reading yet; but from what I understand, it seems like the app just “pushes” its modifications/its own version of the task toward the ICal files that are online, as shown in the functions used in the “sync” function:

private suspend fun pushLocalChanges(
            caldavCalendar: CaldavCalendar, httpClient: OkHttpClient, httpUrl: HttpUrl) {
        for (task in caldavDao.getMoved(caldavCalendar.uuid!!)) {
            deleteRemoteResource(httpClient, httpUrl, task)
        }
        for (task in taskDao.getCaldavTasksToPush(caldavCalendar.uuid!!)) {
            try {
                pushTask(task, httpClient, httpUrl)
            } catch (e: IOException) {
                Timber.e(e)
            }
        }
    }

    private suspend fun deleteRemoteResource(
            httpClient: OkHttpClient, httpUrl: HttpUrl, caldavTask: CaldavTask): Boolean {
        try {
            if (!isNullOrEmpty(caldavTask.`object`)) {
                val remote = DavResource(
                        httpClient, httpUrl.newBuilder().addPathSegment(caldavTask.`object`!!).build())
                remote.delete(null) {}
            }
        } catch (e: HttpException) {
            if (e.code != 404) {
                Timber.e(e)
                return false
            }
        } catch (e: IOException) {
            Timber.e(e)
            return false
        }
        caldavDao.delete(caldavTask)
        return true
    }

    private suspend fun pushTask(task: Task, httpClient: OkHttpClient, httpUrl: HttpUrl) {
        Timber.d("pushing %s", task)
        val caldavTask = caldavDao.getTask(task.id) ?: return
        if (task.isDeleted) {
            if (deleteRemoteResource(httpClient, httpUrl, caldavTask)) {
                taskDeleter.delete(task)
            }
            return
        }
        val data = iCal.toVtodo(caldavTask, task)
        val requestBody = RequestBody.create(MIME_ICALENDAR, data)
        try {
            val remote = DavResource(
                    httpClient, httpUrl.newBuilder().addPathSegment(caldavTask.`object`!!).build())
            remote.put(requestBody) {
                val getETag = fromResponse(it)
                if (getETag != null && !isNullOrEmpty(getETag.eTag)) {
                    caldavTask.etag = getETag.eTag
                    caldavTask.vtodo = String(data)
                }
            }
        } catch (e: HttpException) {
            Timber.e(e)
            return
        }
        caldavTask.lastSync = task.modificationDate
        caldavDao.update(caldavTask)
        Timber.d("SENT %s", caldavTask)
    }

I’ve just tried it myself by creating a task on Nextcloud Tasks, which is synchronized with Tasks .org on my phone. I’ve created a “Test” task, and then synced Tasks .org to receive it in my phone. I then put my phone offline. On Nextcloud task, I wrote “1” as a description of the task; and in Tasks .org, I wrote “2”. Waited a bit, then put my phone online and let Tasks .org synchronize. The result is that the task had a description that said “2”.

What I get from that is that 1) ICal sync rules might not depend on the ICal protocol, but on the software used; and 2) that in the case of Tasks .org, the philosophy seems to be “my recent edits to a task always have the priority on other recent edits made after the last sync elsewhere”. I guess that this could be a good way to do it too ? It means that the a user would have to make an edit to a task in order for a conflict to form, and the client where they made the modification would have the priority.

I made a second experiment to try and see if Tasks .org was able to only “impose” modifications on characteristics of a task that were modified. For example, if I go offline with my phone, and change the due date of a task; while online, the description of the task has been changed; by syncing once my phone gets online, will I get the date entered on my offline phone AND the new description wrote online ? The answer is no. Tasks .org just pushes its own version of the task, all characteristics taken into account. Vikunja could maybe be a bit smarter on this aspect, and allow changes only to characteristics of a task that have been changed while offline.

I hope that this makes sense, and that it could be of some help !

1 Like

The code looks like it’s Kotlin code :slightly_smiling_face:

I think so. That would at least be the easiest to implement, maybe in combination with a timestamp comparison so that it would always use the newest changes regardless of where they were made.

Definitely. To do that kind of comparisons though it would need to save the original task before saving it locally to then be able to compare what fields changed elsewhere.

Maybe we could implement that in multiple stages:

  1. “Dumb push”: changes are saved locally and then pushed once the device gets online again, without any checks if something changed on the server. This follows the way Tasks.org does it.
  2. Before pushing an offline task, check if it was modified on the server. Only use the newer version of the two. That would mean to discard local changes if the server changes are newer and vice-versa.
  3. Check which task fields changed and try to merge them. If merging would lead to a conflict, prompt the user to ask them what changes they want to keep.
1 Like

Sounds awesome ! I’d make one precision though; I think that Tasks.org follows your option 2, meaning that if a task has changed on the server since the last sync - but not on the offline device -, then the offline device will accept the change from the server. I think that this is the most straight forward rule : “if the server changed something and I didn’t, I accept the change; if the server and I both changed something, I push my own change”.

What could be interesting to do (in my own opinion) is to choose your option 3 by default when all is implemented. Then, the first time that a conflict emerge, the user could be treated with a popup asking them which version to keep, but also a checkbox allowing them to switch to option 2 in the future, saying something like:

There is a conflict in your task between your offline and your online version. Please, indicate which version of the task you’d like to keep: [Insert choice]
Don’t warn me of future conflict, and push the local changes onto the server in case of conflict

1 Like

I think that sounds like a good idea.

1 Like

Sorry for the bump; I just wanted to add some experiments I’ve been doing.

Basically, Tasks.org and Dav5x seem to allow for some limited offline changes to the database:

  • You can add a task offline in Tasks.org; when connection is restored, the task will be added to the Vikunja database.
  • You cannot change the name of the tasks offline with Tasks.org; when connection is restored, the change will not be added to the Vikunka database.
  • You can tick tasks offline in Tasks.org; when connection is restored, those will be ticked in the Vikunja database. However, if you changed the name of the task offline in Tasks.org, it won’t be ticked, and the Vikunja database will re-add the task with the correct name in Tasks.org.

So there seems to be some possibilities already, it’s exciting !

1 Like

What about using https://yjs.dev/ as backend for a CRDT based syncing?
There are quite a few apps using CRDTs (and y.js) for p2p offline-first synchronisation

The plan is to use yjs when we’ll tackle offline support. It’s just not on the near future list.

Just some quick note that we haven’t forgotten this yet. It’s just really complex to do it good. To give an example see what I just wrote regarding offline date recognition.

Not exactly what you’re asking for, but I’m working on a completely offline version of Vikunja.

The first step of porting an app to webxdc is to make it fully offline-capable.
I’ll post the updates there.

Eager to see what you’ll come up with.
webxdc sounds interesting although it probably won’t fit everyones usecase.

Do you think you’ll be able to prevent a hard fork?

2 Likes

By the way vlcn looks really interesting. Combined with the fact that the Origin private file system now seems to have quite good support and sqlite itself maintains a web assembly version there are really interesting ways opening up these days :slight_smile:
vlcn uses wa-sqlite though which additionally supports IndexedDB as a fallback.

1 Like

It’s not up to just me, but I’d be very glad to :wink:
I consider what I’m doing right now a prototype. We’ll see how it works out and evaluate whether it can be rewritten in a merge-able way.
Right now it doesn’t look too bad IMO. In short what I did is:

  • Wrote a light /localBackend that supports CRUD for just tasks and buckets (no filtering and sorting). The /services folder is a very narrow place and it’s convenient to cut off the real back-end there and replace it with a local one.
  • Hidden a big portion of the UI (filters, reminders, notifications, comments, most of the settings, auth, the entire sidebar)
  • Removed some logic (like auth, isOnline)

Update: it’s now not just fully offline, but also p2p-syncable through WebRTC, thanks to Yjs. It’s still raw, so consider it a prototype.

2 Likes

Wow that’s even more impressive! Really nice stuff!

3 Likes