refactor

* docs: more additions to chat protocol

* schema wip

* schema

* update diagram

* remove comments

* punctuation

* diagram text

* update threat model

---------

Co-authored-by: Evgeny Poberezkin <2769109+epoberezkin@users.noreply.github.com>

cabal.project

@@ -12,7 +12,7 @@ constraints: zip +disable-bzip2 +disable-zstd
 source-repository-package
     type: git
     location: https://github.com/simplex-chat/simplexmq.git
-    tag: 8a3b72458f917e9867f4e3640dda0fa1827ff6cf
+    tag: 5021f78a7e231fb3764f3bc3865c6902dcaebebc
 
 source-repository-package
     type: git

docs/protocol/diagrams/group.mmd

@@ -4,9 +4,16 @@ sequenceDiagram
   participant B as Bob
   participant C as Existing<br>contact
 
-  note over A, B: 1. send and accept group invitation
-  A ->> B: x.grp.inv<br>invite Bob to group<br>(via contact connection)
-  B ->> A: x.grp.acpt<br>accept invitation<br>(via member connection)<br>establish group member connection
+  alt invite contact
+    note over A, B: 1a. send and accept group invitation
+    A ->> B: x.grp.inv<br>invite Bob to group<br>(via contact connection)
+    B ->> A: x.grp.acpt<br>accept invitation<br>(via member connection)<br>establish group member connection
+  else join via group link
+    note over A, B: 1b. join via group link and accept request
+    B ->> A: join via group link<br>SimpleX contact address
+    A ->> B: x.grp.link.inv in SMP confirmation<br>accept joining member request,<br>sending group profile, etc.<br>establish group member connection
+    A ->> B: x.grp.link.mem<br>send inviting member profile
+  end
 
   note over M, B: 2. introduce new member Bob to all existing members
   A ->> M: x.grp.mem.new<br>"announce" Bob<br>to existing members<br>(via member connections)
@@ -20,14 +27,25 @@ sequenceDiagram
   end
   A ->> M: x.grp.mem.fwd<br>forward "invitations" and<br>Bob's chat protocol version<br>to all members<br>(via member connections)
 
+  note over M, B: group message forwarding<br>(while connections between members are being established)
+  M -->> B: messages between members and Bob are forwarded by Alice
+  B -->> M: 
+
   note over M, B: 3. establish direct and group member connections
   M ->> B: establish group member connection
 
   opt chat protocol compatible version < 2
     M ->> B: establish direct connection
-    note over M, C: 4. deduplicate new contact
+    note over M, C: 3*. deduplicate new contact
     B ->> M: x.info.probe<br>"probe" is sent to all new members
     B ->> C: x.info.probe.check<br>"probe" hash,<br>in case contact and<br>member profiles match
     C ->> B: x.info.probe.ok<br> original "probe",<br> in case contact and member<br>are the same user
     note over B: merge existing and new contacts if received and sent probe hashes match
   end
+
+  note over M, B: 4. notify inviting member that connection is established
+  M ->> A: x.grp.mem.con
+  B ->> A: x.grp.mem.con
+  note over A: stops forwarding messages
+  M -->> B: messages are sent via group connection without forwarding
+  B -->> M: 

docs/protocol/simplex-chat.md

@@ -2,7 +2,7 @@
 title: SimpleX Chat Protocol
 revision: 08.08.2022
 ---
-DRAFT Revision 0.1, 2022-08-08
+Revision 2, 2024-06-24
 
 Evgeny Poberezkin
 
@@ -17,18 +17,23 @@ SimpleX Chat Protocol is a protocol used by SimpleX Chat clients to exchange mes
 The scope of SimpleX Chat Protocol is application level messages, both for chat functionality, related to the conversations between the clients, and extensible for any other application functions. Currently supported chat functions:
 
 - direct and group messages,
-- message replies (quoting), forwarded messages and message deletions,
-- message attachments: images and files,
+- message replies (quoting), message editing, forwarded messages and message deletions,
+- message attachments: images, videos, voice messages and files,
 - creating and managing chat groups,
 - invitation and signalling for audio/video WebRTC calls.
 
 ## General message format
 
-SimpleX Chat protocol supports two message formats:
+SimpleX Chat protocol supports these message formats:
 
 - JSON-based format for chat and application messages.
+- compressed format for adapting larger messages to reduced size of message envelope, caused by addition of PQ encryption keys to SMP agent message envelope.
 - binary format for sending files or any other binary data.
 
+JSON-based message format supports batching inside a single container message, by encoding list of messages as JSON array.
+
+Current implementation of chat protocol in SimpleX Chat uses SimpleX File Transfer Protocol (XFTP) for file transfer, with passing file description as chat protocol messages, instead passing files in binary format via SMP connections.
+
 ### JSON format for chat and application messages
 
 This document uses JTD schemas [RFC 8927](https://www.rfc-editor.org/rfc/rfc8927.html) to define the properties of chat messages, with some additional restrictions on message properties included in metadata member of JTD schemas. In case of any contradiction between JSON examples and JTD schema the latter MUST be considered correct.
@@ -77,8 +82,22 @@ For example, this message defines a simple text message `"hello!"`:
 
 `params` property includes message data, depending on `event`, as defined below and in [JTD schema](./simplex-chat.schema.json).
 
+### Compressed format
+
+The syntax of compressed message is defined by the following ABNF notation:
+
+```abnf
+compressedMessage = %s"X" 1*15780 OCTET; compressed message data
+```
+
+Compressed message is required to fit into 13388 bytes, accounting for agent overhead (see Protocol's maxCompressedMsgLength).
+
+The actual JSON message is required to fit into 15610 bytes, accounting for group message forwarding (x.grp.msg.forward) overhead (see Protocol's maxEncodedMsgLength).
+
 ### Binary format for sending files
 
+> Note: Planned to be deprecated. No longer used for file transfer in SimpleX Chat implementation of chat protocol.
+
 SimpleX Chat clients use separate connections to send files using a binary format. File chunk size send in each message MUST NOT be bigger than 15,780 bytes to fit into 16kb (16384 bytes) transport block.
 
 The syntax of each message used to send files is defined by the following ABNF notation:
@@ -117,7 +136,9 @@ SimpleX Chat Protocol supports the following message types passed in `event` pro
 - `x.contact` - contact profile and additional data sent as part of contact request to a long-term contact address.
 - `x.info*` - messages to send, update and de-duplicate contact profiles.
 - `x.msg.*` - messages to create, update and delete content chat items.
+- `x.msg.file.descr` - message to transfer XFTP file description.
 - `x.file.*` - messages to accept and cancel sending files (see files sub-protocol).
+- `x.direct.del` - message to notify about contact deletion.
 - `x.grp.*` - messages used to manage groups and group members (see group sub-protocol).
 - `x.call.*` - messages to invite to WebRTC calls and send signalling messages.
 - `x.ok` - message sent during connection handshake.
@@ -136,7 +157,7 @@ This message is sent by both sides of the connection during the connection hands
 
 ### Probing for duplicate contacts
 
-As there are no globally unique user identitifiers, when the contact a user is already connected to is added to the group by some other group member, this contact will be added to user's list of contacts as a new contact. To allow merging such contacts, "a probe" (random base64url-encoded 32 bytes) SHOULD be sent to all new members as part of `x.info.probe` message and, in case there is a contact with the same profile, the hash of the probe MAY be sent to it as part of `x.info.probe.check` message. In case both the new member and the existing contact are the same user (they would receive both the probe and its hash), the contact would send back the original probe as part of `x.info.probe.ok` message via the previously existing contact connection – proving to the sender that this new member and the existing contact are the same user, in which case the sender SHOULD merge these two contacts.
+As there are no globally unique user identifiers, when the contact a user is already connected to is added to the group by some other group member, this contact will be added to user's list of contacts as a new contact. To allow merging such contacts, "a probe" (random base64url-encoded 32 bytes) SHOULD be sent to all new members as part of `x.info.probe` message and, in case there is a contact with the same profile, the hash of the probe MAY be sent to it as part of `x.info.probe.check` message. In case both the new member and the existing contact are the same user (they would receive both the probe and its hash), the contact would send back the original probe as part of `x.info.probe.ok` message via the previously existing contact connection – proving to the sender that this new member and the existing contact are the same user, in which case the sender SHOULD merge these two contacts.
 
 Sending clients MAY disable this functionality, and receiving clients MAY ignore probe messages.
 
@@ -155,6 +176,8 @@ Message content can be one of four types:
 - `text` - no file attachment is expected for this format, `text` property MUST be non-empty.
 - `file` - attached file is required, `text` property MAY be empty.
 - `image` - attached file is required, `text` property MAY be empty.
+- `video` - attached file is required, `text` property MAY be empty.
+- `voice` - attached file is required, `text` property MAY be empty.
 - `link` - no file attachment is expected, `text` property MUST be non-empty. `preview` property contains information about link preview.
 
 See `/definition/msgContent` in [JTD schema](./simplex-chat.schema.json) for message container format.
@@ -181,25 +204,29 @@ File attachment can optionally include connection address to receive the file -
 
 `x.file.cancel` message is sent to notify the recipient that sending of the file was cancelled. It is sent in response to accepting the file with `x.file.acpt.inv` message. It is sent in the same connection where the file was offered.
 
+`x.msg.file.descr` message is used to send XFTP file description. File descriptions that don't fit into a single chat protocol message are sent in parts, with messages including part number (`fileDescrPartNo`) and description completion marker (`fileDescrComplete`). Recipient client accumulates description parts and starts file download upon completing file description.
+
 ## Sub-protocol for chat groups
 
 ### Decentralized design for chat groups
 
-SimpleX Chat groups are fully decentralized and do not have any globally unique group identifiers - they are only defined on client devices as a group profile and a set of bi-directional SimpleX connections with other group members. When a new member accepts group invitation, the inviting member introduces a new member to all existing members and forwards the connection addresses so that they can establish direct and group member connections.
+SimpleX Chat groups are fully decentralized and do not have any globally unique group identifiers - they are only defined on client devices as a group profile and a set of bi-directional SimpleX connections with other group members. When a new member accepts group invitation or joins via group link, the inviting member introduces a new member to all existing members and forwards the connection addresses so that they can establish direct and group member connections.
 
 There is a possibility of the attack here: as the introducing member forwards the addresses, they can substitute them with other addresses, performing MITM attack on the communication between existing and introduced members - this is similar to the communication operator being able to perform MITM on any connection between the users. To mitigate this attack this group sub-protocol will be extended to allow validating security of the connection by sending connection verification out-of-band.
 
-Clients are RECOMMENDED to indicate in the UI whether the connection to a group member or contact was made directly or via annother user.
+Clients are RECOMMENDED to indicate in the UI whether the connection to a group member or contact was made directly or via another user.
 
 Each member in the group is identified by a group-wide unique identifier used by all members in the group. This is to allow referencing members in the messages and to allow group message integrity validation.
 
 The diagram below shows the sequence of messages sent between the users' clients to add the new member to the group.
 
+While introduced members establish connection inside group, inviting member forwards messages between them by sending `x.grp.msg.forward` messages. When introduced members finalize connection, they notify inviting member to stop forwarding via `x.grp.mem.con` message.
+
 ![Adding member to the group](./diagrams/group.svg)
 
 ### Member roles
 
-Currently members can have one of three roles - `owner`, `admin` and `member`. The user that created the group is self-assigned owner role, the new members are assigned role by the member who adds them - only `owner` and `admin` members can add new members; only `owner` members can add members with `owner` role.
+Currently members can have one of three roles - `owner`, `admin`, `member` and `observer`. The user that created the group is self-assigned owner role, the new members are assigned role by the member who adds them - only `owner` and `admin` members can add new members; only `owner` members can add members with `owner` role. `Observer` members only receive messages and aren't allowed to send messages.
 
 ### Messages to manage groups and add members
 
@@ -207,6 +234,10 @@ Currently members can have one of three roles - `owner`, `admin` and `member`. T
 
 `x.grp.acpt` message is sent as part of group member connection handshake, only to the inviting user.
 
+`x.grp.link.inv` message is sent as part of connection handshake to member joining via group link, and contains group profile and initial information about inviting and joining member.
+
+`x.grp.link.mem` message is sent as part of connection handshake to member joining via group link, and contains remaining information about inviting member.
+
 `x.grp.mem.new` message is sent by the inviting user to all connected members (and scheduled as pending to all announced but not yet connected members) to announce a new member to the existing members. This message MUST only be sent by members with `admin` or `owner` role. Receiving clients MUST ignore this message if it is received from member with `member` role.
 
 `x.grp.mem.intro` messages are sent by the inviting user to the invited member, via their group member connection, one message for each existing member. When this message is sent by any other member than the one who invited the recipient it MUST be ignored.
@@ -219,6 +250,10 @@ Currently members can have one of three roles - `owner`, `admin` and `member`. T
 
 `x.grp.mem.role` message is sent to update group member role - it is sent to all members by the member who updated the role of the member referenced in this message. This message MUST only be sent by members with `admin` or `owner` role. Receiving clients MUST ignore this message if it is received from member with role less than `admin`.
 
+`x.grp.mem.restrict` message is sent to group members to communicate group member restrictions, such as member being blocked for sending messages.
+
+`x.grp.mem.con` message is sent by members connecting inside group to inviting member, to notify the inviting member they have completed the connection and no longer require forwarding messages between them.
+
 `x.grp.mem.del` message is sent to delete a member - it is sent to all members by the member who deletes the member referenced in this message. This message MUST only be sent by members with `admin` or `owner` role. Receiving clients MUST ignore this message if it is received from member with `member` role.
 
 `x.grp.leave` message is sent to all members by the member leaving the group. If the only group `owner` leaves the group, it will not be possible to delete it with `x.grp.del` message - but all members can still leave the group with `x.grp.leave` message and then delete a local copy of the group.
@@ -227,6 +262,10 @@ Currently members can have one of three roles - `owner`, `admin` and `member`. T
 
 `x.grp.info` message is sent to all members by the member who updated group profile. Only group owners can update group profiles. Clients MAY implement some conflict resolution strategy - it is currently not implemented by SimpleX Chat client. This message MUST only be sent by members with `owner` role. Receiving clients MUST ignore this message if it is received from member other than with `owner` role.
 
+`x.grp.direct.inv` message is sent to a group member to propose establishing a direct connection between members, thus creating a contact with another member.
+
+`x.grp.msg.forward` message is sent by inviting member to forward messages between introduced members, while they are connecting.
+
 ## Sub-protocol for WebRTC audio/video calls
 
 This sub-protocol is used to send call invitations and to negotiate end-to-end encryption keys and pass WebRTC signalling information.
@@ -240,3 +279,66 @@ These message are used for WebRTC calls:
 3. `x.call.answer`: to continue with call connection the initiating clients must reply with `x.call.answer` message. This message contains WebRTC answer and collected ICE candidates. Additional ICE candidates can be sent in `x.call.extra` message.
 
 4. `x.call.end` message is sent to notify the other party that the call is terminated.
+
+## Threat model
+
+This threat model compliments SMP, XFTP, push notifications and XRCP protocols threat models:
+
+- [SimpleX Messaging Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/overview-tjr.md#threat-model);
+- [SimpleX File Transfer Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/xftp.md#threat-model);
+- [Push notifications threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/push-notifications.md#threat-model);
+- [SimpleX Remote Control Protocol threat model](https://github.com/simplex-chat/simplexmq/blob/master/protocol/xrcp.md#threat-model).
+
+#### A user's contact
+
+*can:*
+
+- send messages prohibited by user's preferences or otherwise act non-compliantly with user's preferences (for example, if message with updated preferences was lost or failed to be processed, or with modified client), in which case user client should treat such messages and actions as prohibited.
+
+- by exchanging special messages with user's client, match user's contact with existing group members and/or contacts that have identical user profile (see [Probing for duplicate contacts](#probing-for-duplicate-contacts)).
+
+- identify that and when a user is using SimpleX, in case user has delivery receipts enabled, or based on other automated client responses.
+
+*cannot:*
+
+- match user's contact with existing group members and/or contacts with different or with incognito profiles.
+
+- match user's contact without communicating with the user's client.
+
+#### A group member
+
+*can:*
+
+- send messages prohibited by group's preferences and member restrictions or otherwise act non-compliantly with preferences and restrictions (for example, if decentralized group state diverged, or with modified client), in which case user client should treat such messages and actions as prohibited.
+
+- create a direct contact with a user if group permissions allow it.
+
+- by exchanging special messages with user's client, match user's group member record with the existing group members and/or contacts that have identical user profile.
+
+- undetectably send different messages to different group members, or selectively send messages to some members and not send to others.
+
+- identify that and when a user is using SimpleX, in case user has delivery receipts enabled, or based on other automated client responses.
+
+- join the same group several times, from the same or from different user profile, and pretend to be different members.
+
+*cannot:*
+
+- match user's contact with existing group members and/or contacts with different or with incognito profiles.
+
+- match user's group member record with existing group members and/or contacts without communication of user's client.
+
+- determine whether two group members with different or with incognito profiles are the same user.
+
+#### A group admin
+
+*can:*
+
+- carry out MITM attack between user and other group member(s) when forwarding invitations for group connections (user can detect such attack by verifying connection security codes out-of-band).
+
+- undetectably forward different messages to different group members, selectively adding, modifying, and dropping forwarded messages.
+
+- disrupt decentralized group state by sending different messages that change group state (such as adding or removing members, member role changes, etc.) to different group members, or sending such messages selectively.
+
+*cannot:*
+
+- prove that two group members with incognito profiles is the same user.

docs/protocol/simplex-chat.schema.json

@@ -8,7 +8,7 @@
         "displayName": {
           "type": "string",
           "metadata": {
-            "format": "non-empty string without spaces, the first character must not be # or @"
+            "format": "non-empty string, the first character must not be # or @"
           }
         },
         "fullName": {"type": "string"}
@@ -19,6 +19,39 @@
           "metadata": {
             "format": "data URI format for base64 encoded image"
           }
+        },
+        "contactLink": {"ref": "connReqUri"},
+        "preferences": {
+          "type": "string",
+          "metadata": {
+            "format": "JSON encoded user preferences"
+          }
+        }
+      },
+      "additionalProperties": true
+    },
+    "groupProfile": {
+      "properties": {
+        "displayName": {
+          "type": "string",
+          "metadata": {
+            "format": "non-empty string, the first character must not be # or @"
+          }
+        },
+        "fullName": {"type": "string"}
+      },
+      "optionalProperties": {
+        "image": {
+          "type": "string",
+          "metadata": {
+            "format": "data URI format for base64 encoded image"
+          }
+        },
+        "groupPreferences": {
+          "type": "string",
+          "metadata": {
+            "format": "JSON encoded user preferences"
+          }
         }
       },
       "additionalProperties": true
@@ -29,6 +62,8 @@
       },
       "optionalProperties": {
         "file": {"ref": "fileInvitation"},
+        "ttl": {"type": "integer"},
+        "live": {"type": "boolean"},
         "quote": {
           "properties": {
             "msgRef": {"ref": "msgRef"},
@@ -56,17 +91,47 @@
           }
         },
         "image": {
-          "text": {"type": "string", "metadata": {"comment": "can be empty"}},
-          "image": {"ref": "base64url"}
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "image": {"ref": "base64url"}
+          }
+        },
+        "video": {
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "image": {"ref": "base64url"},
+            "duration": {"type": "integer"}
+          }
+        },
+        "voice": {
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}},
+            "duration": {"type": "integer"}
+          }
         },
         "file": {
-          "text": {"type": "string", "metadata": {"comment": "can be empty"}}
+          "properties": {
+            "text": {"type": "string", "metadata": {"comment": "can be empty"}}
+          }
         }
       },
       "metadata": {
         "comment": "it is RECOMMENDED that the clients support other values in `type` properties showing them as text messages in case `text` property is present"
       }
     },
+    "msgReaction" : {
+      "discriminator": "type",
+      "mapping": {
+        "emoji": {
+          "properties": {
+            "emoji": {
+              "type": "string",
+              "metadata": {"comment": "emoji character"}
+            }
+          }
+        }
+      }
+    },
     "msgRef": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -91,7 +156,31 @@
         "fileSize": {"type": "uint32"}
       },
       "optionalProperties": {
-        "fileConnReq": {"ref": "connReqUri"}
+        "fileDigest": {"ref": "base64url"},
+        "fileConnReq": {"ref": "connReqUri"},
+        "fileDescr": {"ref": "fileDescription"}
+      }
+    },
+    "fileDescription": {
+      "properties": {
+        "fileDescrText": {
+          "type": "string",
+          "metadata": {
+            "format": "XFTP file description part text"
+          }
+        },
+        "fileDescrPartNo": {
+          "type": "integer",
+          "metadata": {
+            "format": "XFTP file description part number"
+          }
+        },
+        "fileDescrComplete": {
+          "type": "boolean",
+          "metadata": {
+            "format": "XFTP file description completion marker"
+          }
+        }
       }
     },
     "linkPreview": {
@@ -100,6 +189,21 @@
         "title": {"type": "string"},
         "description": {"type": "string"},
         "image": {"ref": "base64url"}
+      },
+      "optionalProperties": {
+        "content": {"ref": "linkContent"}
+      }
+    },
+    "linkContent": {
+      "discriminator": "type",
+      "mapping": {
+        "page": {},
+        "image": {},
+        "video": {
+          "optionalProperties": {
+            "duration": {"type": "integer"}
+          }
+        }
       }
     },
     "groupInvitation": {
@@ -107,15 +211,27 @@
         "fromMember": {"ref": "memberIdRole"},
         "invitedMember": {"ref": "memberIdRole"},
         "connRequest": {"ref": "connReqUri"},
-        "groupProfile": {"ref": "profile"}
+        "groupProfile": {"ref": "groupProfile"}
       },
       "optionalProperties": {
         "groupLinkId": {"ref": "base64url"},
+        "groupSize": {"type": "integer"},
         "metadata": {
-          "comment": "used to identify invitation via group link"
+          "comment": "groupLinkId is used to identify invitation via group link"
         }
       }
     },
+    "groupLinkInvitation": {
+      "properties": {
+        "fromMember": {"ref": "memberIdRole"},
+        "fromMemberName": {"type": "string"},
+        "invitedMember": {"ref": "memberIdRole"},
+        "groupProfile": {"ref": "groupProfile"}
+      },
+      "optionalProperties": {
+        "groupSize": {"type": "integer"}
+      }
+    },
     "memberIdRole": {
       "properties": {
         "memberId": {"ref": "base64url"},
@@ -127,16 +243,35 @@
         "memberId": {"ref": "base64url"},
         "memberRole": {"ref": "groupMemberRole"},
         "profile": {"ref": "profile"}
+      },
+      "optionalProperties": {
+        "v": {"ref": "chatVersionRange"}
+      }
+    },
+    "memberRestrictions": {
+      "properties": {
+        "restriction": {"ref": "memberRestrictionStatus"}
+      }
+    },
+    "memberRestrictionStatus": {
+      "enum": ["blocked", "unrestricted"]
+    },
+    "chatVersionRange": {
+      "type": "string",
+      "metadata": {
+        "format": "chat version range string encoded as `<min>-<max>`, or as `<number>` if min = max"
       }
     },
     "introInvitation": {
       "properties": {
-        "groupConnReq": {"ref": "connReqUri"},
+        "groupConnReq": {"ref": "connReqUri"}
+      },
+      "optionalProperties": {
         "directConnReq": {"ref": "connReqUri"}
       }
     },
     "groupMemberRole": {
-      "enum": ["author", "member", "admin", "owner"]
+      "enum": ["observer", "author", "member", "admin", "owner"]
     },
     "callInvitation": {
       "properties": {
@@ -257,6 +392,17 @@
         "params": {"ref": "msgContainer"}
       }
     },
+    "x.msg.file.descr": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "msgId": {"ref": "base64url"},
+            "fileDescr": {"ref": "fileDescription"}
+          }
+        }
+      }
+    },
     "x.msg.update": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -264,6 +410,10 @@
           "properties": {
             "msgId": {"ref": "base64url"},
             "content": {"ref": "msgContent"}
+          },
+          "optionalProperties": {
+            "ttl": {"type": "integer"},
+            "live": {"type": "boolean"}
           }
         }
       }
@@ -274,6 +424,24 @@
         "params": {
           "properties": {
             "msgId": {"ref": "base64url"}
+          },
+          "optionalProperties": {
+            "memberId": {"ref": "base64url"}
+          }
+        }
+      }
+    },
+    "x.msg.react": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "msgId": {"ref": "base64url"},
+            "reaction": {"ref": "msgReaction"},
+            "add": {"type": "boolean"}
+          },
+          "optionalProperties": {
+            "memberId": {"ref": "base64url"}
           }
         }
       }
@@ -294,8 +462,10 @@
         "params": {
           "properties": {
             "msgId": {"ref": "base64url"},
-            "fileConnReq": {"ref": "connReqUri"},
             "fileName": {"type": "string"}
+          },
+          "optionalProperties": {
+            "fileConnReq": {"ref": "connReqUri"}
           }
         }
       }
@@ -310,6 +480,14 @@
         }
       }
     },
+    "x.direct.del": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {}
+        }
+      }
+    },
     "x.grp.inv": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -330,6 +508,26 @@
         }
       }
     },
+    "x.grp.link.inv": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "groupLinkInvitation": {"ref": "groupLinkInvitation"}
+          }
+        }
+      }
+    },
+    "x.grp.link.mem": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "profile": {"ref": "profile"}
+          }
+        }
+      }
+    },
     "x.grp.mem.new": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -346,6 +544,9 @@
         "params": {
           "properties": {
             "memberInfo": {"ref": "memberInfo"}
+          },
+          "optionalProperties": {
+            "memberRestrictions": {"ref": "memberRestrictions"}
           }
         }
       }
@@ -394,6 +595,27 @@
         }
       }
     },
+    "x.grp.mem.restrict": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"},
+            "memberRestrictions": {"ref": "memberRestrictions"}
+          }
+        }
+      }
+    },
+    "x.grp.mem.con": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"}
+          }
+        }
+      }
+    },
     "x.grp.mem.del": {
       "properties": {
         "msgId": {"ref": "base64url"},
@@ -425,7 +647,42 @@
         "msgId": {"ref": "base64url"},
         "params": {
           "properties": {
-            "groupProfile": {"ref": "profile"}
+            "groupProfile": {"ref": "groupProfile"}
+          }
+        }
+      }
+    },
+    "x.grp.direct.inv": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "connReq": {"ref": "connReqUri"}
+          },
+          "optionalProperties": {
+            "content": {"ref": "msgContent"}
+          }
+        }
+      }
+    },
+    "x.grp.msg.forward": {
+      "properties": {
+        "msgId": {"ref": "base64url"},
+        "params": {
+          "properties": {
+            "memberId": {"ref": "base64url"},
+            "msg": {
+              "type": "string",
+              "metadata": {
+                "format": "JSON encoded chat message"
+              }
+            },
+            "msgTs": {
+              "type": "string",
+              "metadata": {
+                "format": "ISO8601 UTC time of the message"
+              }
+            }
           }
         }
       }
@@ -436,7 +693,7 @@
         "params": {
           "properties": {
             "callId": {"ref": "base64url"},
-            "invitation": {}
+            "invitation": {"ref": "callInvitation"}
           }
         }
       }

docs/rfcs/2024-06-17-agent-stats-persistence.md

@@ -0,0 +1,87 @@
+# Agent stats persistence
+
+## Problem
+
+State/state tracked in agent are lost on app restart, which makes it difficult to debug user bugs.
+
+## Solution
+
+Persist stats between sessions.
+
+App terminal signals may vary per platform / be absent (?) -> persist stats periodically.
+
+Stats would have `<userId, server>` key, so we don't want to store them in a plaintext file to not leak used servers locally -> persist in encrypted db.
+
+There's couple of orthogonal design decision to be made:
+- persist in chat or in agent db
+  - pros for chat:
+    - possibly less contention for db than agent
+  - pros for agent:
+    - no unnecessary back and forth, especially if agent starts accumulating from past sessions and has to be parameterized with past stats (see below)
+- agent to start accumulating from past sessions stats, or keep past separately and only accumulate for current session from zeros
+  - pros for accumulating from past sessions:
+    - easier to maintain stats - e.g. user deletion has to remove keys, which is more convoluted if past stats are not stored in memory
+    - simpler UI - overall stats, no differentiation for past/current session (or less logic in backend preparing presentation data)
+  - pros for accumulating from zeros:
+    - simpler start logic - no need to restore stats from agent db / pass initial stats from chat db
+    - can differentiate between past sessions and current session stats in UI
+
+### Option 1 - Persist in chat db, agent to track only current session
+
+- Chat stores stats in such table:
+
+```sql
+CREATE TABLE agent_stats(
+  agent_stats_id INTEGER PRIMARY KEY, -- dummy id, there will only be one record
+  past_stats TEXT, -- accumulated from previous sessions
+  session_stats TEXT, -- current session
+  past_started_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of tracking stats, reset on stats reset
+  session_started_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of current session
+  session_updated_at TEXT NOT NULL DEFAULT(datetime('now')) -- last update of current session stats (periodic, frequent updates)
+);
+```
+
+- Chat periodically calls getAgentServersStats api and updates `session_stats`.
+  - interval? should be short to not lose too much data, 5-30 seconds?
+- On start `session_stats` are accumulated into `past_stats` and set to null.
+- On user deletion, agent updates current session stats in memory (removes keys), chat has to do same for both stats fields in db.
+  - other cases where stats have to be manipulated in similar way?
+
+### Option 2 - Persist in chat db, agent to accumulate stats from past sessions
+
+- Table is only used for persistence of overall stats:
+
+```sql
+CREATE TABLE agent_stats(
+  agent_stats_id INTEGER PRIMARY KEY, -- dummy id, there will only be one record
+  agent_stats TEXT, -- overall stats - past and session
+  started_tracking_at TEXT NOT NULL DEFAULT(datetime('now')), -- starting point of tracking stats, reset on stats reset
+  updated_at TEXT NOT NULL DEFAULT(datetime('now'))
+);
+```
+
+- Chat to parameterize creation of agent client with initial stats.
+
+### Option 3 - Persist in agent db, agent to differentiate past stats and session stats
+
+- Table in agent db similar to option 1.
+- Agent is responsible for periodic updates in session, as well as accumulating into "past" and resetting session stats on start.
+- Agent only communicates stats to chat on request.
+- On user deletion agent is fully responsible for maintaining both in-memory session stats, and updating db records.
+
+### Option 4 - Persist in agent db, agent to accumulate stats from past sessions
+
+- Table in agent db similar to option 2.
+- On start agent restores initial stats into memory by itself.
+- Since all stats are in memory, on user deletion it's enough to update in memory without updating db.
+  - there is a race possible where agent crashes after updating stats (removing user keys) in memory before database stats have been overwritten by a periodic update, so it may be better to immediately overwrite and not wait for periodic update.
+  - still at least there's at least no additional logic to update past stats.
+
+### Other considerations
+
+Why is it important to timely remove user keys from past stats?
+- stats not being saved for past users:
+  - important both privacy-wise and to not cause confusion when showing "All" stats (e.g. user summing up across users stats would have smaller total than total stats).
+  - to avoid accidentally mixing up with newer users.
+    - though we do have an AUTOINCREMENT user_id in agent so probably it wouldn't be a problem.
+- on the other hand maybe we don't want to "forget" stats on user deletion so that stats would reflect networking more accurately?

scripts/nix/sha256map.nix

@@ -1,5 +1,5 @@
 {
-  "https://github.com/simplex-chat/simplexmq.git"."8a3b72458f917e9867f4e3640dda0fa1827ff6cf" = "1mmxdaj563kjmlkacxdnq62n6mzw9khampzaqghnk6iiwzdig0qy";
+  "https://github.com/simplex-chat/simplexmq.git"."5021f78a7e231fb3764f3bc3865c6902dcaebebc" = "0yrcdc07q86kaw35y3pk52chgmxhv5yc348wd2fw89p7lfqcgwwy";
   "https://github.com/simplex-chat/hs-socks.git"."a30cc7a79a08d8108316094f8f2f82a0c5e1ac51" = "0yasvnr7g91k76mjkamvzab2kvlb1g5pspjyjn2fr6v83swjhj38";
   "https://github.com/simplex-chat/direct-sqlcipher.git"."f814ee68b16a9447fbb467ccc8f29bdd3546bfd9" = "1ql13f4kfwkbaq7nygkxgw84213i0zm7c1a8hwvramayxl38dq5d";
   "https://github.com/simplex-chat/sqlcipher-simple.git"."a46bd361a19376c5211f1058908fc0ae6bf42446" = "1z0r78d8f0812kxbgsm735qf6xx8lvaz27k1a0b4a2m0sshpd5gl";

simplex-chat.cabal

@@ -160,6 +160,7 @@ library
       Simplex.Chat.Remote.RevHTTP
       Simplex.Chat.Remote.Transport
       Simplex.Chat.Remote.Types
+      Simplex.Chat.Stats
       Simplex.Chat.Store
       Simplex.Chat.Store.AppSettings
       Simplex.Chat.Store.Connections

src/Simplex/Chat.hs

@@ -71,6 +71,7 @@ import Simplex.Chat.ProfileGenerator (generateRandomProfile)
 import Simplex.Chat.Protocol
 import Simplex.Chat.Remote
 import Simplex.Chat.Remote.Types
+import Simplex.Chat.Stats
 import Simplex.Chat.Store
 import Simplex.Chat.Store.AppSettings
 import Simplex.Chat.Store.Connections
@@ -84,7 +85,6 @@ import Simplex.Chat.Store.Shared
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Chat.Util (encryptFile, liftIOEither, shuffle)
 import qualified Simplex.Chat.Util as U
 import Simplex.FileTransfer.Client.Main (maxFileSize, maxFileSizeHard)
@@ -95,7 +95,7 @@ import Simplex.FileTransfer.Protocol (FileParty (..), FilePartyI)
 import qualified Simplex.FileTransfer.Transport as XFTP
 import Simplex.FileTransfer.Types (FileErrorType (..), RcvFileId, SndFileId)
 import Simplex.Messaging.Agent as Agent
-import Simplex.Messaging.Agent.Client (AgentStatsKey (..), SubInfo (..), agentClientStore, getAgentQueuesInfo, getAgentWorkersDetails, getAgentWorkersSummary, getNetworkConfig', ipAddressProtected, withLockMap)
+import Simplex.Messaging.Agent.Client (AgentStatsKey (..), SMPServerSubs (..), SubInfo (..), agentClientStore, getAgentQueuesInfo, getAgentWorkersDetails, getAgentWorkersSummary, getNetworkConfig', ipAddressProtected, withLockMap)
 import Simplex.Messaging.Agent.Env.SQLite (AgentConfig (..), InitialAgentServers (..), createAgentStore, defaultAgentConfig)
 import Simplex.Messaging.Agent.Lock (withLock)
 import Simplex.Messaging.Agent.Protocol
@@ -113,7 +113,7 @@ import qualified Simplex.Messaging.Crypto.Ratchet as CR
 import Simplex.Messaging.Encoding
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (base64P)
-import Simplex.Messaging.Protocol (AProtoServerWithAuth (..), AProtocolType (..), EntityId, ErrorType (..), MsgBody, MsgFlags (..), NtfServer, ProtoServerWithAuth (..), ProtocolTypeI, SProtocolType (..), SubscriptionMode (..), UserProtocol, XFTPServer, userProtocol)
+import Simplex.Messaging.Protocol (AProtoServerWithAuth (..), AProtocolType (..), EntityId, ErrorType (..), MsgBody, MsgFlags (..), NtfServer, ProtoServerWithAuth (..), ProtocolServer, ProtocolTypeI, SProtocolType (..), SubscriptionMode (..), UserProtocol, XFTPServer, userProtocol)
 import qualified Simplex.Messaging.Protocol as SMP
 import Simplex.Messaging.ServiceScheme (ServiceScheme (..))
 import qualified Simplex.Messaging.TMap as TM
@@ -309,12 +309,12 @@ newChatController
       agentServers :: ChatConfig -> IO InitialAgentServers
       agentServers config@ChatConfig {defaultServers = defServers@DefaultAgentServers {ntf, netCfg}} = do
         users <- withTransaction chatStore getUsers
-        smp' <- getUserServers users SPSMP
-        xftp' <- getUserServers users SPXFTP
+        smp' <- getUsersServers users SPSMP
+        xftp' <- getUsersServers users SPXFTP
         pure InitialAgentServers {smp = smp', xftp = xftp', ntf, netCfg}
         where
-          getUserServers :: forall p. (ProtocolTypeI p, UserProtocol p) => [User] -> SProtocolType p -> IO (Map UserId (NonEmpty (ProtoServerWithAuth p)))
-          getUserServers users protocol = case users of
+          getUsersServers :: forall p. (ProtocolTypeI p, UserProtocol p) => [User] -> SProtocolType p -> IO (Map UserId (NonEmpty (ProtoServerWithAuth p)))
+          getUsersServers users protocol = case users of
             [] -> pure $ M.fromList [(1, cfgServers protocol defServers)]
             _ -> M.fromList <$> initialServers
             where
@@ -369,7 +369,7 @@ activeAgentServers ChatConfig {defaultServers} p =
   fromMaybe (cfgServers p defaultServers)
     . nonEmpty
     . map (\ServerCfg {server} -> server)
-    . filter (\ServerCfg {enabled} -> enabled)
+    . filter (\ServerCfg {enabled} -> enabled == SEEnabled)
 
 cfgServers :: UserProtocol p => SProtocolType p -> (DefaultAgentServers -> NonEmpty (ProtoServerWithAuth p))
 cfgServers p DefaultAgentServers {smp, xftp} = case p of
@@ -1309,19 +1309,21 @@ processChatCommand' vr = \case
         withStore (\db -> Just <$> getConnectionEntity db vr user agentConnId) `catchChatError` (\e -> toView (CRChatError (Just user) e) $> Nothing)
     pure CRNtfMessages {user_, connEntity_, msgTs = msgTs', ntfMessages = map ntfMsgInfo msgs}
   APIGetUserProtoServers userId (AProtocolType p) -> withUserId userId $ \user -> withServerProtocol p $ do
-    ChatConfig {defaultServers} <- asks config
-    servers <- withStore' (`getProtocolServers` user)
-    let defServers = cfgServers p defaultServers
-        servers' = fromMaybe (L.map toServerCfg defServers) $ nonEmpty servers
-    pure $ CRUserProtoServers user $ AUPS $ UserProtoServers p servers' defServers
-    where
-      toServerCfg server = ServerCfg {server, preset = True, tested = Nothing, enabled = True}
+    (srvCfgs, defServers) <- getUserServers user p
+    pure $ CRUserProtoServers user $ AUPS $ UserProtoServers p srvCfgs defServers
   GetUserProtoServers aProtocol -> withUser $ \User {userId} ->
     processChatCommand $ APIGetUserProtoServers userId aProtocol
-  APISetUserProtoServers userId (APSC p (ProtoServersConfig servers)) -> withUserId userId $ \user -> withServerProtocol p $ do
-    withStore $ \db -> overwriteProtocolServers db user servers
+  APISetUserProtoServers userId (APSC p (ProtoServersConfig srvsCfg)) -> withUserId userId $ \user -> withServerProtocol p $ do
+    withStore $ \db -> overwriteProtocolServers db user srvsCfg
     cfg <- asks config
-    lift $ withAgent' $ \a -> setProtocolServers a (aUserId user) $ activeAgentServers cfg p servers
+    lift $ withAgent' $ \a -> setProtocolServers a (aUserId user) $ activeAgentServers cfg p srvsCfg
+    ok user
+  APIAddKnownProtoServer userId (AProtoServerWithAuth p newSrv) -> withUserId userId $ \user -> withServerProtocol p $ do
+    (srvCfgs, _) <- getUserServers user p
+    when (protoServer newSrv `elem` map (\ServerCfg {server} -> protoServer server) (L.toList srvCfgs)) $
+      throwChatError (CECommandError "server already configured")
+    let newSrvCfg = ServerCfg {server = newSrv, preset = False, tested = Nothing, enabled = SEKnown}
+    withStore $ \db -> overwriteProtocolServers db user (L.toList srvCfgs <> [newSrvCfg])
     ok user
   SetUserProtoServers serversConfig -> withUser $ \User {userId} ->
     processChatCommand $ APISetUserProtoServers userId serversConfig
@@ -2253,6 +2255,31 @@ processChatCommand' vr = \case
         CLUserContact ucId -> "UserContact " <> show ucId
         CLFile fId -> "File " <> show fId
   DebugEvent event -> toView event >> ok_
+  GetAgentServersSummary userId -> withUserId userId $ \user -> do
+    agentServersSummary <- lift $ withAgent' getAgentServersSummary
+    users <- withStore' getUsers
+    smpServers <- getUserSrvs user SPSMP
+    xftpServers <- getUserSrvs user SPXFTP
+    let presentedServersSummary = toPresentedServersSummary agentServersSummary users user smpServers xftpServers
+    pure $ CRAgentServersSummary user presentedServersSummary
+    where
+      getUserSrvs :: forall p. (ProtocolTypeI p, UserProtocol p) => User -> SProtocolType p -> CM [ProtocolServer p]
+      getUserSrvs user protocol = do
+        (srvCfgs, _) <- getUserServers user protocol
+        pure $ map (\ServerCfg {server} -> protoServer server) (L.toList srvCfgs)
+  ResetAgentServersStats -> withAgent resetAgentServersStats >> ok_
+  GetAgentSubsSummary userId -> withUserId userId $ \user -> do
+    agentSubsSummary <- lift $ withAgent' getAgentSubsSummary
+    users <- withStore' getUsers
+    let subs = countSubs agentSubsSummary user users
+    pure $ CRAgentSubsSummary user subs
+    where
+      countSubs agentSubsSummary user users = do
+        M.foldrWithKey' addSubs SMPServerSubs {ssActive = 0, ssPending = 0} agentSubsSummary
+        where
+          addSubs auId subs subsAcc
+            | countUserInAllStats (AgentUserId auId) user users = addSMPSubs subs subsAcc
+            | otherwise = subsAcc
   GetAgentWorkers -> lift $ CRAgentWorkersSummary <$> withAgent' getAgentWorkersSummary
   GetAgentWorkersDetails -> lift $ CRAgentWorkersDetails <$> withAgent' getAgentWorkersDetails
   GetAgentStats -> lift $ CRAgentStats . map stat <$> withAgent' getAgentStats
@@ -2830,6 +2857,16 @@ processChatCommand' vr = \case
       msgInfo <- withStore' (`getLastRcvMsgInfo` connId)
       CRQueueInfo user msgInfo <$> withAgent (`getConnectionQueueInfo` acId)
 
+getUserServers :: forall p. (ProtocolTypeI p, UserProtocol p) => User -> SProtocolType p -> CM (NonEmpty (ServerCfg p), NonEmpty (ProtoServerWithAuth p))
+getUserServers user p = do
+  ChatConfig {defaultServers} <- asks config
+  servers <- withStore' (`getProtocolServers` user)
+  let defServers = cfgServers p defaultServers
+      userServers = fromMaybe (L.map defServerCfg defServers) $ nonEmpty servers
+  pure (userServers, defServers)
+  where
+    defServerCfg server = ServerCfg {server, preset = True, tested = Nothing, enabled = SEEnabled}
+
 contactCITimed :: Contact -> CM (Maybe CITimed)
 contactCITimed ct = sndContactCITimed False ct Nothing
 
@@ -3214,10 +3251,8 @@ receiveViaCompleteFD user fileId RcvFileDescr {fileDescrText, fileDescrComplete}
       S.toList $ S.fromList $ concatMap (\FD.FileChunk {replicas} -> map (\FD.FileChunkReplica {server} -> server) replicas) chunks
     getUnknownSrvs :: [XFTPServer] -> CM [XFTPServer]
     getUnknownSrvs srvs = do
-      ChatConfig {defaultServers = DefaultAgentServers {xftp = defXftp}} <- asks config
-      storedSrvs <- map (\ServerCfg {server} -> protoServer server) <$> withStore' (`getProtocolServers` user)
-      let defXftp' = L.map protoServer defXftp
-          knownSrvs = fromMaybe defXftp' $ nonEmpty storedSrvs
+      (srvCfgs, _) <- getUserServers user SPXFTP
+      let knownSrvs = map (\ServerCfg {server} -> protoServer server) (L.toList srvCfgs)
       pure $ filter (`notElem` knownSrvs) srvs
     ipProtectedForSrvs :: [XFTPServer] -> CM Bool
     ipProtectedForSrvs srvs = do
@@ -7413,6 +7448,7 @@ chatCommandP =
       "/xftp test " *> (TestProtoServer . AProtoServerWithAuth SPXFTP <$> strP),
       "/ntf test " *> (TestProtoServer . AProtoServerWithAuth SPNTF <$> strP),
       "/_servers " *> (APISetUserProtoServers <$> A.decimal <* A.space <*> srvCfgP),
+      "/_known server " *> (APIAddKnownProtoServer <$> A.decimal <* A.space <*> strP),
       "/smp " *> (SetUserProtoServers . APSC SPSMP . ProtoServersConfig . map toServerCfg <$> protocolServersP),
       "/smp default" $> SetUserProtoServers (APSC SPSMP $ ProtoServersConfig []),
       "/xftp " *> (SetUserProtoServers . APSC SPXFTP . ProtoServersConfig . map toServerCfg <$> protocolServersP),
@@ -7611,6 +7647,9 @@ chatCommandP =
       ("/version" <|> "/v") $> ShowVersion,
       "/debug locks" $> DebugLocks,
       "/debug event " *> (DebugEvent <$> jsonP),
+      "/get servers summary " *> (GetAgentServersSummary <$> A.decimal),
+      "/reset servers stats" $> ResetAgentServersStats,
+      "/get subs summary " *> (GetAgentSubsSummary <$> A.decimal),
       "/get stats" $> GetAgentStats,
       "/reset stats" $> ResetAgentStats,
       "/get subs" $> GetAgentSubs,
@@ -7755,7 +7794,7 @@ chatCommandP =
         (Just <$> (AutoAccept <$> (" incognito=" *> onOffP <|> pure False) <*> optional (A.space *> msgContentP)))
         (pure Nothing)
     srvCfgP = strP >>= \case AProtocolType p -> APSC p <$> (A.space *> jsonP)
-    toServerCfg server = ServerCfg {server, preset = False, tested = Nothing, enabled = True}
+    toServerCfg server = ServerCfg {server, preset = False, tested = Nothing, enabled = SEEnabled}
     rcCtrlAddressP = RCCtrlAddress <$> ("addr=" *> strP) <*> (" iface=" *> (jsonP <|> text1P))
     text1P = safeDecodeUtf8 <$> A.takeTill (== ' ')
     char_ = optional . A.char

src/Simplex/Chat/Call.hs

@@ -21,10 +21,10 @@ import Data.Time.Clock (UTCTime)
 import Database.SQLite.Simple.FromField (FromField (..))
 import Database.SQLite.Simple.ToField (ToField (..))
 import Simplex.Chat.Types (Contact, ContactId, User)
-import Simplex.Chat.Types.Util (decodeJSON, encodeJSON)
 import qualified Simplex.Messaging.Crypto as C
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fromTextField_, fstToLower, singleFieldJSON)
+import Simplex.Messaging.Util (decodeJSON, encodeJSON)
 
 data Call = Call
   { contactId :: ContactId,

src/Simplex/Chat/Controller.hs

@@ -60,6 +60,7 @@ import Simplex.Chat.Messages.CIContent
 import Simplex.Chat.Protocol
 import Simplex.Chat.Remote.AppVersion
 import Simplex.Chat.Remote.Types
+import Simplex.Chat.Stats (PresentedServersSummary)
 import Simplex.Chat.Store (AutoAccept, ChatLockEntity, StoreError (..), UserContactLink, UserMsgReceiptSettings)
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
@@ -68,14 +69,14 @@ import Simplex.Chat.Types.UITheme
 import Simplex.Chat.Util (liftIOEither)
 import Simplex.FileTransfer.Description (FileDescriptionURI)
 import Simplex.Messaging.Agent (AgentClient, SubscriptionsInfo)
-import Simplex.Messaging.Agent.Client (AgentLocks, AgentQueuesInfo (..), AgentWorkersDetails (..), AgentWorkersSummary (..), ProtocolTestFailure, UserNetworkInfo)
+import Simplex.Messaging.Agent.Client (AgentLocks, AgentQueuesInfo (..), AgentWorkersDetails (..), AgentWorkersSummary (..), ProtocolTestFailure, SMPServerSubs, UserNetworkInfo)
 import Simplex.Messaging.Agent.Env.SQLite (AgentConfig, NetworkConfig)
 import Simplex.Messaging.Agent.Lock
 import Simplex.Messaging.Agent.Protocol
 import Simplex.Messaging.Agent.Store.SQLite (MigrationConfirmation, SQLiteStore, UpMigration, withTransaction)
 import Simplex.Messaging.Agent.Store.SQLite.DB (SlowQueryStats (..))
 import qualified Simplex.Messaging.Agent.Store.SQLite.DB as DB
-import Simplex.Messaging.Client (SMPProxyMode (..), SMPProxyFallback (..))
+import Simplex.Messaging.Client (SMPProxyFallback (..), SMPProxyMode (..))
 import qualified Simplex.Messaging.Crypto as C
 import Simplex.Messaging.Crypto.File (CryptoFile (..))
 import qualified Simplex.Messaging.Crypto.File as CF
@@ -343,6 +344,7 @@ data ChatCommand
   | APIGetUserProtoServers UserId AProtocolType
   | GetUserProtoServers AProtocolType
   | APISetUserProtoServers UserId AProtoServersConfig
+  | APIAddKnownProtoServer UserId AProtoServerWithAuth
   | SetUserProtoServers AProtoServersConfig
   | APITestProtoServer UserId AProtoServerWithAuth
   | TestProtoServer AProtoServerWithAuth
@@ -505,6 +507,9 @@ data ChatCommand
   | ShowVersion
   | DebugLocks
   | DebugEvent ChatResponse
+  | GetAgentServersSummary UserId
+  | ResetAgentServersStats
+  | GetAgentSubsSummary UserId
   | GetAgentStats
   | ResetAgentStats
   | GetAgentSubs
@@ -756,6 +761,8 @@ data ChatResponse
   | CRSQLResult {rows :: [Text]}
   | CRSlowSQLQueries {chatQueries :: [SlowSQLQuery], agentQueries :: [SlowSQLQuery]}
   | CRDebugLocks {chatLockName :: Maybe String, chatEntityLocks :: Map String String, agentLocks :: AgentLocks}
+  | CRAgentServersSummary {user :: User, serversSummary :: PresentedServersSummary}
+  | CRAgentSubsSummary {user :: User, subsSummary :: SMPServerSubs}
   | CRAgentStats {agentStats :: [[String]]}
   | CRAgentWorkersDetails {agentWorkersDetails :: AgentWorkersDetails}
   | CRAgentWorkersSummary {agentWorkersSummary :: AgentWorkersSummary}

src/Simplex/Chat/Markdown.hs

@@ -29,13 +29,12 @@ import Data.Text (Text)
 import qualified Data.Text as T
 import Data.Text.Encoding (encodeUtf8)
 import Simplex.Chat.Types
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (AConnectionRequestUri (..), ConnReqUriData (..), ConnectionRequestUri (..), SMPQueue (..))
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fstToLower, sumTypeJSON)
 import Simplex.Messaging.Protocol (ProtocolServer (..))
 import Simplex.Messaging.ServiceScheme (ServiceScheme (..))
-import Simplex.Messaging.Util (safeDecodeUtf8)
+import Simplex.Messaging.Util (decodeJSON, safeDecodeUtf8)
 import System.Console.ANSI.Types
 import qualified Text.Email.Validate as Email
 
@@ -146,7 +145,7 @@ parseMarkdown s = fromRight (unmarked s) $ A.parseOnly (markdownP <* A.endOfInpu
 
 isSimplexLink :: Format -> Bool
 isSimplexLink = \case
-  SimplexLink {} -> True;
+  SimplexLink {} -> True
   _ -> False
 
 markdownP :: Parser Markdown

src/Simplex/Chat/Messages/CIContent.hs

@@ -29,12 +29,11 @@ import Simplex.Chat.Protocol
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (MsgErrorType (..), RatchetSyncState (..), SwitchPhase (..))
-import Simplex.Messaging.Crypto.Ratchet (PQEncryption, pattern PQEncOn, pattern PQEncOff)
+import Simplex.Messaging.Crypto.Ratchet (PQEncryption, pattern PQEncOff, pattern PQEncOn)
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fstToLower, singleFieldJSON, sumTypeJSON)
-import Simplex.Messaging.Util (safeDecodeUtf8, tshow, (<$?>))
+import Simplex.Messaging.Util (encodeJSON, safeDecodeUtf8, tshow, (<$?>))
 
 data MsgDirection = MDRcv | MDSnd
   deriving (Eq, Show)

src/Simplex/Chat/Protocol.hs

@@ -46,14 +46,13 @@ import Database.SQLite.Simple.ToField (ToField (..))
 import Simplex.Chat.Call
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Shared
-import Simplex.Chat.Types.Util
 import Simplex.Messaging.Agent.Protocol (VersionSMPA, pqdrSMPAgentVersion)
 import Simplex.Messaging.Compression (Compressed, compress1, decompress1)
 import Simplex.Messaging.Encoding
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, fromTextField_, fstToLower, parseAll, sumTypeJSON, taggedObjectJSON)
 import Simplex.Messaging.Protocol (MsgBody)
-import Simplex.Messaging.Util (eitherToMaybe, safeDecodeUtf8, (<$?>))
+import Simplex.Messaging.Util (decodeJSON, eitherToMaybe, encodeJSON, safeDecodeUtf8, (<$?>))
 import Simplex.Messaging.Version hiding (version)
 
 -- Chat version history:

src/Simplex/Chat/Stats.hs

@@ -0,0 +1,275 @@
+{-# LANGUAGE DuplicateRecordFields #-}
+{-# LANGUAGE NamedFieldPuns #-}
+{-# LANGUAGE TemplateHaskell #-}
+{-# OPTIONS_GHC -fno-warn-ambiguous-fields #-}
+
+module Simplex.Chat.Stats where
+
+import qualified Data.Aeson.TH as J
+import Data.Map.Strict (Map)
+import qualified Data.Map.Strict as M
+import Data.Maybe (fromMaybe, isJust)
+import Data.Time.Clock (UTCTime)
+import Simplex.Chat.Types
+import Simplex.Messaging.Agent.Client
+import Simplex.Messaging.Agent.Protocol (UserId)
+import Simplex.Messaging.Agent.Stats
+import Simplex.Messaging.Parsers (defaultJSON)
+import Simplex.Messaging.Protocol
+
+data PresentedServersSummary = PresentedServersSummary
+  { statsStartedAt :: UTCTime,
+    currentUserServers :: ServersSummary,
+    allUsersServers :: ServersSummary
+  }
+  deriving (Show)
+
+-- Presentation of servers will be split into separate categories,
+-- so users can differentiate currently used (connected) servers,
+-- previously connected servers that were in use in previous sessions,
+-- and servers that are only proxied (not connected directly).
+data ServersSummary = ServersSummary
+  { -- currently used SMP servers are those with Just in sessions and/or subs in SMPServerSummary;
+    -- all other servers would fall either into previously used or only proxied servers category
+    currentlyUsedSMPServers :: [SMPServerSummary],
+    -- previously used SMP servers are those with Nothing in sessions and subs,
+    -- and have any of sentDirect, sentProxied, recvMsgs, etc. > 0 in server stats (see toPresentedServersSummary);
+    -- remaining servers would fall into only proxied servers category
+    previouslyUsedSMPServers :: [SMPServerSummary],
+    -- only proxied SMP servers are those that aren't (according to current state - sessions and subs)
+    -- and weren't (according to stats) connected directly; they would have Nothing in sessions and subs,
+    -- and have all of sentDirect, sentProxied, recvMsgs, etc. = 0 in server stats
+    onlyProxiedSMPServers :: [SMPServerSummary],
+    -- currently used XFTP servers are those with Just in sessions in XFTPServerSummary,
+    -- and/or have upload/download/deletion in progress;
+    -- all other servers would fall into previously used servers category
+    currentlyUsedXFTPServers :: [XFTPServerSummary],
+    -- previously used XFTP servers are those with Nothing in sessions and don't have any process in progress
+    previouslyUsedXFTPServers :: [XFTPServerSummary]
+  }
+  deriving (Show)
+
+data SMPServerSummary = SMPServerSummary
+  { smpServer :: SMPServer,
+    -- known:
+    -- for simplicity always Nothing in totalServersSummary - allows us to load configured servers only for current user,
+    -- and also unnecessary unless we want to add navigation to other users servers settings;
+    -- always Just in currentUserServers - True if server is in list of user servers, otherwise False;
+    -- True - allows to navigate to server settings, False - allows to add server to configured as known (SEKnown)
+    known :: Maybe Bool,
+    sessions :: Maybe ServerSessions,
+    subs :: Maybe SMPServerSubs,
+    -- stats:
+    -- even if sessions and subs are Nothing, stats can be Just - server could be used earlier in session,
+    -- or in previous sessions and stats for it were restored; server would fall into a category of
+    -- previously used or only proxied servers - see ServersSummary above
+    stats :: Maybe AgentSMPServerStatsData
+  }
+  deriving (Show)
+
+data XFTPServerSummary = XFTPServerSummary
+  { xftpServer :: XFTPServer,
+    known :: Maybe Bool, -- same as for SMPServerSummary
+    sessions :: Maybe ServerSessions,
+    stats :: Maybe AgentXFTPServerStatsData,
+    rcvInProgress :: Bool,
+    sndInProgress :: Bool,
+    delInProgress :: Bool
+  }
+  deriving (Show)
+
+-- Maps AgentServersSummary to PresentedServersSummary:
+-- - currentUserServers is for currentUser;
+-- - users are passed to exclude hidden users from totalServersSummary;
+-- - if currentUser is hidden, it should be accounted in totalServersSummary;
+-- - known is set only in user level summaries based on passed userSMPSrvs and userXFTPSrvs
+toPresentedServersSummary :: AgentServersSummary -> [User] -> User -> [SMPServer] -> [XFTPServer] -> PresentedServersSummary
+toPresentedServersSummary agentSummary users currentUser userSMPSrvs userXFTPSrvs = do
+  let (userSMPSrvsSumms, allSMPSrvsSumms) = accSMPSrvsSummaries
+      (userSMPCurr, userSMPPrev, userSMPProx) = smpSummsIntoCategories userSMPSrvsSumms
+      (allSMPCurr, allSMPPrev, allSMPProx) = smpSummsIntoCategories allSMPSrvsSumms
+      (userXFTPSrvsSumms, allXFTPSrvsSumms) = accXFTPSrvsSummaries
+      (userXFTPCurr, userXFTPPrev) = xftpSummsIntoCategories userXFTPSrvsSumms
+      (allXFTPCurr, allXFTPPrev) = xftpSummsIntoCategories allXFTPSrvsSumms
+  PresentedServersSummary
+    { statsStartedAt,
+      currentUserServers =
+        ServersSummary
+          { currentlyUsedSMPServers = userSMPCurr,
+            previouslyUsedSMPServers = userSMPPrev,
+            onlyProxiedSMPServers = userSMPProx,
+            currentlyUsedXFTPServers = userXFTPCurr,
+            previouslyUsedXFTPServers = userXFTPPrev
+          },
+      allUsersServers =
+        ServersSummary
+          { currentlyUsedSMPServers = allSMPCurr,
+            previouslyUsedSMPServers = allSMPPrev,
+            onlyProxiedSMPServers = allSMPProx,
+            currentlyUsedXFTPServers = allXFTPCurr,
+            previouslyUsedXFTPServers = allXFTPPrev
+          }
+    }
+  where
+    AgentServersSummary {statsStartedAt, smpServersSessions, smpServersSubs, smpServersStats, xftpServersSessions, xftpServersStats, xftpRcvInProgress, xftpSndInProgress, xftpDelInProgress} = agentSummary
+    countUserInAll auId = countUserInAllStats (AgentUserId auId) currentUser users
+    smpSummsIntoCategories :: Map SMPServer SMPServerSummary -> ([SMPServerSummary], [SMPServerSummary], [SMPServerSummary])
+    smpSummsIntoCategories = foldr partitionSummary ([], [], [])
+      where
+        partitionSummary srvSumm (curr, prev, prox)
+          | isCurrentlyUsed srvSumm = (srvSumm : curr, prev, prox)
+          | isPreviouslyUsed srvSumm = (curr, srvSumm : prev, prox)
+          | otherwise = (curr, prev, srvSumm : prox)
+        isCurrentlyUsed SMPServerSummary {sessions, subs} = isJust sessions || isJust subs
+        isPreviouslyUsed SMPServerSummary {stats} = case stats of
+          Nothing -> False
+          -- add connCompleted, connDeleted?
+          -- check: should connCompleted be counted for proxy? is it?
+          Just AgentSMPServerStatsData {_sentDirect, _sentProxied, _sentDirectAttempts, _sentProxiedAttempts, _recvMsgs, _connCreated, _connSecured, _connSubscribed, _connSubAttempts} ->
+            _sentDirect > 0 || _sentProxied > 0 || _sentDirectAttempts > 0 || _sentProxiedAttempts > 0 || _recvMsgs > 0 || _connCreated > 0 || _connSecured > 0 || _connSubscribed > 0 || _connSubAttempts > 0
+    xftpSummsIntoCategories :: Map XFTPServer XFTPServerSummary -> ([XFTPServerSummary], [XFTPServerSummary])
+    xftpSummsIntoCategories = foldr partitionSummary ([], [])
+      where
+        partitionSummary srvSumm (curr, prev)
+          | isCurrentlyUsed srvSumm = (srvSumm : curr, prev)
+          | otherwise = (curr, srvSumm : prev)
+        isCurrentlyUsed XFTPServerSummary {sessions, rcvInProgress, sndInProgress, delInProgress} =
+          isJust sessions || rcvInProgress || sndInProgress || delInProgress
+    accSMPSrvsSummaries :: (Map SMPServer SMPServerSummary, Map SMPServer SMPServerSummary)
+    accSMPSrvsSummaries = M.foldrWithKey' (addServerData addStats) summs2 smpServersStats
+      where
+        summs1 = M.foldrWithKey' (addServerData addSessions) (M.empty, M.empty) smpServersSessions
+        summs2 = M.foldrWithKey' (addServerData addSubs) summs1 smpServersSubs
+        addServerData ::
+          (a -> SMPServerSummary -> SMPServerSummary) ->
+          (UserId, SMPServer) ->
+          a ->
+          (Map SMPServer SMPServerSummary, Map SMPServer SMPServerSummary) ->
+          (Map SMPServer SMPServerSummary, Map SMPServer SMPServerSummary)
+        addServerData addData (userId, srv) d (userSumms, allUsersSumms) = (userSumms', allUsersSumms')
+          where
+            userSumms'
+              | userId == aUserId currentUser = alterSumms newUserSummary userSumms
+              | otherwise = userSumms
+            allUsersSumms'
+              | countUserInAll userId = alterSumms newSummary allUsersSumms
+              | otherwise = allUsersSumms
+            alterSumms n = M.alter (Just . addData d . fromMaybe n) srv
+            newUserSummary = (newSummary :: SMPServerSummary) {known = Just $ srv `elem` userSMPSrvs}
+            newSummary =
+              SMPServerSummary
+                { smpServer = srv,
+                  known = Nothing,
+                  sessions = Nothing,
+                  subs = Nothing,
+                  stats = Nothing
+                }
+        addSessions :: ServerSessions -> SMPServerSummary -> SMPServerSummary
+        addSessions s summ@SMPServerSummary {sessions} = summ {sessions = Just $ maybe s (s `addServerSessions`) sessions}
+        addSubs :: SMPServerSubs -> SMPServerSummary -> SMPServerSummary
+        addSubs s summ@SMPServerSummary {subs} = summ {subs = Just $ maybe s (s `addSMPSubs`) subs}
+        addStats :: AgentSMPServerStatsData -> SMPServerSummary -> SMPServerSummary
+        addStats s summ@SMPServerSummary {stats} = summ {stats = Just $ maybe s (s `addSMPStats`) stats}
+    accXFTPSrvsSummaries :: (Map XFTPServer XFTPServerSummary, Map XFTPServer XFTPServerSummary)
+    accXFTPSrvsSummaries = M.foldrWithKey' (addServerData addStats) summs1 xftpServersStats
+      where
+        summs1 = M.foldrWithKey' (addServerData addSessions) (M.empty, M.empty) xftpServersSessions
+        addServerData ::
+          (a -> XFTPServerSummary -> XFTPServerSummary) ->
+          (UserId, XFTPServer) ->
+          a ->
+          (Map XFTPServer XFTPServerSummary, Map XFTPServer XFTPServerSummary) ->
+          (Map XFTPServer XFTPServerSummary, Map XFTPServer XFTPServerSummary)
+        addServerData addData (userId, srv) d (userSumms, allUsersSumms) = (userSumms', allUsersSumms')
+          where
+            userSumms'
+              | userId == aUserId currentUser = alterSumms newUserSummary userSumms
+              | otherwise = userSumms
+            allUsersSumms'
+              | countUserInAll userId = alterSumms newSummary allUsersSumms
+              | otherwise = allUsersSumms
+            alterSumms n = M.alter (Just . addData d . fromMaybe n) srv
+            newUserSummary = (newSummary :: XFTPServerSummary) {known = Just $ srv `elem` userXFTPSrvs}
+            newSummary =
+              XFTPServerSummary
+                { xftpServer = srv,
+                  known = Nothing,
+                  sessions = Nothing,
+                  stats = Nothing,
+                  rcvInProgress = srv `elem` xftpRcvInProgress,
+                  sndInProgress = srv `elem` xftpSndInProgress,
+                  delInProgress = srv `elem` xftpDelInProgress
+                }
+        addSessions :: ServerSessions -> XFTPServerSummary -> XFTPServerSummary
+        addSessions s summ@XFTPServerSummary {sessions} = summ {sessions = Just $ maybe s (s `addServerSessions`) sessions}
+        addStats :: AgentXFTPServerStatsData -> XFTPServerSummary -> XFTPServerSummary
+        addStats s summ@XFTPServerSummary {stats} = summ {stats = Just $ maybe s (s `addXFTPStats`) stats}
+    addServerSessions :: ServerSessions -> ServerSessions -> ServerSessions
+    addServerSessions ss1 ss2 =
+      ServerSessions
+        { ssConnected = ssConnected ss1 + ssConnected ss2,
+          ssErrors = ssErrors ss1 + ssErrors ss2,
+          ssConnecting = ssConnecting ss1 + ssConnecting ss2
+        }
+    addSMPStats :: AgentSMPServerStatsData -> AgentSMPServerStatsData -> AgentSMPServerStatsData
+    addSMPStats sd1 sd2 =
+      AgentSMPServerStatsData
+        { _sentDirect = _sentDirect sd1 + _sentDirect sd2,
+          _sentViaProxy = _sentViaProxy sd1 + _sentViaProxy sd2,
+          _sentProxied = _sentProxied sd1 + _sentProxied sd2,
+          _sentDirectAttempts = _sentDirectAttempts sd1 + _sentDirectAttempts sd2,
+          _sentViaProxyAttempts = _sentViaProxyAttempts sd1 + _sentViaProxyAttempts sd2,
+          _sentProxiedAttempts = _sentProxiedAttempts sd1 + _sentProxiedAttempts sd2,
+          _sentAuthErrs = _sentAuthErrs sd1 + _sentAuthErrs sd2,
+          _sentQuotaErrs = _sentQuotaErrs sd1 + _sentQuotaErrs sd2,
+          _sentExpiredErrs = _sentExpiredErrs sd1 + _sentExpiredErrs sd2,
+          _sentOtherErrs = _sentOtherErrs sd1 + _sentOtherErrs sd2,
+          _recvMsgs = _recvMsgs sd1 + _recvMsgs sd2,
+          _recvDuplicates = _recvDuplicates sd1 + _recvDuplicates sd2,
+          _recvCryptoErrs = _recvCryptoErrs sd1 + _recvCryptoErrs sd2,
+          _recvErrs = _recvErrs sd1 + _recvErrs sd2,
+          _ackMsgs = _ackMsgs sd1 + _ackMsgs sd2,
+          _ackAttempts = _ackAttempts sd1 + _ackAttempts sd2,
+          _connCreated = _connCreated sd1 + _connCreated sd2,
+          _connSecured = _connSecured sd1 + _connSecured sd2,
+          _connCompleted = _connCompleted sd1 + _connCompleted sd2,
+          _connDeleted = _connDeleted sd1 + _connDeleted sd2,
+          _connSubscribed = _connSubscribed sd1 + _connSubscribed sd2,
+          _connSubAttempts = _connSubAttempts sd1 + _connSubAttempts sd2,
+          _connSubErrs = _connSubErrs sd1 + _connSubErrs sd2
+        }
+    addXFTPStats :: AgentXFTPServerStatsData -> AgentXFTPServerStatsData -> AgentXFTPServerStatsData
+    addXFTPStats sd1 sd2 =
+      AgentXFTPServerStatsData
+        { _uploads = _uploads sd1 + _uploads sd2,
+          _uploadAttempts = _uploadAttempts sd1 + _uploadAttempts sd2,
+          _uploadErrs = _uploadErrs sd1 + _uploadErrs sd2,
+          _downloads = _downloads sd1 + _downloads sd2,
+          _downloadAttempts = _downloadAttempts sd1 + _downloadAttempts sd2,
+          _downloadAuthErrs = _downloadAuthErrs sd1 + _downloadAuthErrs sd2,
+          _downloadErrs = _downloadErrs sd1 + _downloadErrs sd2,
+          _deletions = _deletions sd1 + _deletions sd2,
+          _deleteAttempts = _deleteAttempts sd1 + _deleteAttempts sd2,
+          _deleteErrs = _deleteErrs sd1 + _deleteErrs sd2
+        }
+
+countUserInAllStats :: AgentUserId -> User -> [User] -> Bool
+countUserInAllStats (AgentUserId auId) currentUser users =
+  auId == aUserId currentUser || auId `notElem` hiddenUserIds
+  where
+    hiddenUserIds = map aUserId $ filter (isJust . viewPwdHash) users
+
+addSMPSubs :: SMPServerSubs -> SMPServerSubs -> SMPServerSubs
+addSMPSubs ss1 ss2 =
+  SMPServerSubs
+    { ssActive = ssActive ss1 + ssActive ss2,
+      ssPending = ssPending ss1 + ssPending ss2
+    }
+
+$(J.deriveJSON defaultJSON ''SMPServerSummary)
+
+$(J.deriveJSON defaultJSON ''XFTPServerSummary)
+
+$(J.deriveJSON defaultJSON ''ServersSummary)
+
+$(J.deriveJSON defaultJSON ''PresentedServersSummary)

src/Simplex/Chat/Store/Profiles.hs

@@ -523,9 +523,10 @@ getProtocolServers db User {userId} =
       (userId, decodeLatin1 $ strEncode protocol)
   where
     protocol = protocolTypeI @p
-    toServerCfg :: (NonEmpty TransportHost, String, C.KeyHash, Maybe Text, Bool, Maybe Bool, Bool) -> ServerCfg p
-    toServerCfg (host, port, keyHash, auth_, preset, tested, enabled) =
+    toServerCfg :: (NonEmpty TransportHost, String, C.KeyHash, Maybe Text, Bool, Maybe Bool, Int) -> ServerCfg p
+    toServerCfg (host, port, keyHash, auth_, preset, tested, enabledInt) =
       let server = ProtoServerWithAuth (ProtocolServer protocol host port keyHash) (BasicAuth . encodeUtf8 <$> auth_)
+          enabled = toServerEnabled enabledInt
        in ServerCfg {server, preset, tested, enabled}
 
 overwriteProtocolServers :: forall p. ProtocolTypeI p => DB.Connection -> User -> [ServerCfg p] -> ExceptT StoreError IO ()
@@ -542,7 +543,7 @@ overwriteProtocolServers db User {userId} servers =
             (protocol, host, port, key_hash, basic_auth, preset, tested, enabled, user_id, created_at, updated_at)
           VALUES (?,?,?,?,?,?,?,?,?,?,?)
         |]
-        ((protocol, host, port, keyHash, safeDecodeUtf8 . unBasicAuth <$> auth_) :. (preset, tested, enabled, userId, currentTs, currentTs))
+        ((protocol, host, port, keyHash, safeDecodeUtf8 . unBasicAuth <$> auth_) :. (preset, tested, fromServerEnabled enabled, userId, currentTs, currentTs))
     pure $ Right ()
   where
     protocol = decodeLatin1 $ strEncode $ protocolTypeI @p

src/Simplex/Chat/Types.hs

@@ -1632,10 +1632,42 @@ data ServerCfg p = ServerCfg
   { server :: ProtoServerWithAuth p,
     preset :: Bool,
     tested :: Maybe Bool,
-    enabled :: Bool
+    enabled :: ServerEnabled
   }
   deriving (Show)
 
+data ServerEnabled
+  = SEDisabled
+  | SEEnabled
+  | -- server is marked as known, but it's not in the list of configured servers;
+    -- e.g., it may be added via an unknown server dialogue and user didn't manually configure it,
+    -- meaning server wasn't tested (or at least such option wasn't presented in UI)
+    -- and it may be inoperable for user due to server password
+    SEKnown
+  deriving (Eq, Show)
+
+pattern DBSEDisabled :: Int
+pattern DBSEDisabled = 0
+
+pattern DBSEEnabled :: Int
+pattern DBSEEnabled = 1
+
+pattern DBSEKnown :: Int
+pattern DBSEKnown = 2
+
+toServerEnabled :: Int -> ServerEnabled
+toServerEnabled = \case
+  DBSEDisabled -> SEDisabled
+  DBSEEnabled -> SEEnabled
+  DBSEKnown -> SEKnown
+  _ -> SEDisabled
+
+fromServerEnabled :: ServerEnabled -> Int
+fromServerEnabled = \case
+  SEDisabled -> DBSEDisabled
+  SEEnabled -> DBSEEnabled
+  SEKnown -> DBSEKnown
+
 data ChatVersion
 
 instance VersionScope ChatVersion
@@ -1764,6 +1796,8 @@ $(JQ.deriveJSON defaultJSON ''ContactRef)
 
 $(JQ.deriveJSON defaultJSON ''NoteFolder)
 
+$(JQ.deriveJSON (enumJSON $ dropPrefix "SE") ''ServerEnabled)
+
 instance ProtocolTypeI p => ToJSON (ServerCfg p) where
   toEncoding = $(JQ.mkToEncoding defaultJSON ''ServerCfg)
   toJSON = $(JQ.mkToJSON defaultJSON ''ServerCfg)

src/Simplex/Chat/Types/Preferences.hs

@@ -36,7 +36,7 @@ import Simplex.Chat.Types.Shared
 import Simplex.Chat.Types.Util
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fromTextField_, sumTypeJSON)
-import Simplex.Messaging.Util (safeDecodeUtf8, (<$?>))
+import Simplex.Messaging.Util (decodeJSON, encodeJSON, safeDecodeUtf8, (<$?>))
 
 data ChatFeature
   = CFTimedMessages

src/Simplex/Chat/Types/UITheme.hs

@@ -18,6 +18,7 @@ import Database.SQLite.Simple.ToField (ToField (..))
 import Simplex.Chat.Types.Util
 import Simplex.Messaging.Encoding.String
 import Simplex.Messaging.Parsers (defaultJSON, dropPrefix, enumJSON, fromTextField_)
+import Simplex.Messaging.Util (decodeJSON, encodeJSON)
 
 data UITheme = UITheme
   { themeId :: Text,

src/Simplex/Chat/Types/Util.hs

@@ -2,26 +2,15 @@
 
 module Simplex.Chat.Types.Util where
 
-import Data.Aeson (FromJSON, ToJSON)
 import qualified Data.Aeson as J
 import qualified Data.Aeson.Types as JT
 import Data.ByteString (ByteString)
-import qualified Data.ByteString.Lazy.Char8 as LB
-import Data.Text (Text)
-import Data.Text.Encoding (encodeUtf8)
 import Data.Typeable
 import Database.SQLite.Simple (ResultError (..), SQLData (..))
 import Database.SQLite.Simple.FromField (FieldParser, returnError)
 import Database.SQLite.Simple.Internal (Field (..))
 import Database.SQLite.Simple.Ok (Ok (Ok))
 import Simplex.Messaging.Encoding.String
-import Simplex.Messaging.Util (safeDecodeUtf8)
-
-encodeJSON :: ToJSON a => a -> Text
-encodeJSON = safeDecodeUtf8 . LB.toStrict . J.encode
-
-decodeJSON :: FromJSON a => Text -> Maybe a
-decodeJSON = J.decode . LB.fromStrict . encodeUtf8
 
 textParseJSON :: TextEncoding a => String -> J.Value -> JT.Parser a
 textParseJSON name = J.withText name $ maybe (fail $ "bad " <> name) pure . textDecode

src/Simplex/Chat/View.hs

@@ -51,10 +51,9 @@ import Simplex.Chat.Types.Preferences
 import Simplex.Chat.Types.Shared
 import Simplex.Chat.Types.UITheme
 import qualified Simplex.FileTransfer.Transport as XFTP
-import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), SubscriptionsInfo (..))
+import Simplex.Messaging.Agent.Client (ProtocolTestFailure (..), ProtocolTestStep (..), SMPServerSubs (..), SubscriptionsInfo (..))
 import Simplex.Messaging.Agent.Env.SQLite (NetworkConfig (..))
 import Simplex.Messaging.Agent.Protocol
-import Simplex.Messaging.Agent.Protocol (AgentErrorType (RCP))
 import Simplex.Messaging.Agent.Store.SQLite.DB (SlowQueryStats (..))
 import Simplex.Messaging.Client (SMPProxyFallback, SMPProxyMode (..))
 import qualified Simplex.Messaging.Crypto as C
@@ -366,6 +365,8 @@ responseToView hu@(currentRH, user_) ChatConfig {logLevel, showReactions, showRe
       "chat entity locks: " <> viewJSON chatEntityLocks,
       "agent locks: " <> viewJSON agentLocks
     ]
+  CRAgentServersSummary u serversSummary -> ttyUser u ["agent servers summary: " <> viewJSON serversSummary]
+  CRAgentSubsSummary u SMPServerSubs {ssActive, ssPending} -> ttyUser u ["agent subscriptions summary: active = " <> sShow ssActive <> ", pending = " <> sShow ssPending]
   CRAgentStats stats -> map (plain . intercalate ",") stats
   CRAgentSubs {activeSubs, pendingSubs, removedSubs} ->
     [plain $ "Subscriptions: active = " <> show (sum activeSubs) <> ", pending = " <> show (sum pendingSubs) <> ", removed = " <> show (sum $ M.map length removedSubs)]

tests/ChatTests/Profiles.hs

@@ -17,7 +17,7 @@ import Simplex.Chat.Store.Shared (createContact)
 import Simplex.Chat.Types (ConnStatus (..), Profile (..))
 import Simplex.Chat.Types.Shared (GroupMemberRole (..))
 import Simplex.Chat.Types.UITheme
-import Simplex.Chat.Types.Util (encodeJSON)
+import Simplex.Messaging.Util (encodeJSON)
 import Simplex.Messaging.Encoding.String (StrEncoding (..))
 import System.Directory (copyFile, createDirectoryIfMissing)
 import Test.Hspec hiding (it)