blade/simplex-chat
1
0
Fork 0
mirror of https://github.com/simplex-chat/simplex-chat.git synced 2024-12-17 17:20:21 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into remote-desktop

Browse Source
This commit is contained in:
Evgeny Poberezkin
2023-11-01 18:05:51 +00:00
parent 02c0cd5619 4cc20a2d32
commit b2f9270452
61 changed files with 2235 additions and 654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfcs/2023-09-25-groups-integrity.md
+124
View File
@@ -0,0 +1,124 @@
# Groups integrity
## Problems
- Inconsistency of group state:
- group profile including group wide preferences,
- list of members and their roles.
- Lack of group messages integrity - group member can send different messages to different members.
Lack of group consistency leads to group federation both in terms of members list and content visible to different members, which leads to user frustration and lack of trust.
Improvements to group design should provide:
- Consistent group state.
- Group messages integrity:
- integrity violations (different message sent to different members) should be identified and shown to users,
- missed messages should be requested to fill in gaps.
## Design ideas and questions
### Group messages integrity
A message container to include member's message ID (ordered?), and list of IDs and hashes of parent messages.
```haskell
data MsgParentId = MsgParentId
{ memberId :: MemberId,
msgId :: Int64, -- sequential message ID for parent message (among memberId member messages)
msgHash :: ByteString
}
data MsgIds = MsgIds
{ msgId :: Int64, -- sequential message ID for member's message
parentIds :: [MsgParentId]
}
```
Questions:
- What level of protocol should include MsgIds, and what messages should be included into integrity graph?
- Having it on AppMessage level would allow to include all protocol messages. But some protocol messages are sent with different content per member (XGrpMemIntro, XGrpMemFwd, probe messages) and would have different hash. Also they contain sensitive data such as invitation links and should not be forwarded anyway.
- If MsgIds is MsgContainer level, only XMsgNew would have it. This excludes other content messages such as updates, deletes, etc.
- Include it into specific "content" chat events - XMsgNew, XMsgFileCancel (unused), XMsgUpdate, XMsgDel, XMsgReact, XFile (not used anymore but was never fully deprecated), XFileCancel.
- Some new protocol level container, uniting above events?
- Should msgId be sequential integer? (It leaks metadata about member's previous activity in the group) Can SharedMsgId be used instead?
- Depending on number of parent messages, parentIds can become arbitrarily long and not fit into 16KB block, especially for messages containing profiles pictures.
When receiving a message with unknown parent identifiers, client should request missing messages from the sender by sending XGrpRequestSkipped, including last seen message reference for each missing parent. When receiving XGrpRequestSkipped, member should forward requested messages up to last seen parent using XGrpRequested.
```haskell
-- include received parentId?
XGrpRequestSkipped :: [MsgParentId] -> ChatMsgEvent 'Json
data MsgRequestedParent = MsgRequested
{ parentId :: MsgParentId,
msg :: MsgContainer -- content TBD based on scope of messages included into integrity graph. Full event?
}
XGrpRequested :: MsgRequestedParent -> ChatMsgEvent 'Json
```
Questions:
- Depending on number of missing parents, XGrpRequestSkipped may not fit into 16KB block.
- There may be multiple skipped messages for a given member, should they be sent sequentially from oldest (following the one known to requesting member) to newest?
- XGrpRequested may not fit into 16KB block even if original MsgContainer / chat event did fit. On the other hand multiple XGrpRequested messages can be batched.
- Malicious group member may arbitrarily request (at any time or in response to a new message) any number of skipped messages by sending parentIds from the past and trigger receiving member to send a lot of traffic. There are already some automatic response events in protocol, but they are harder to abuse: XGrpMemFwd - requires cooperation with other member, or creating connection; receipts - can be turned off; probes - requires member having matching contact and being non incognito in group. Should the member receiving XGrpRequestSkipped protect from such abuse by limiting number of requested messages? Limiting number or requests from a specific member in time?
- By the time member requests skipped messages, sender may be offline. Should the requester send XGrpRequestSkipped to other members?
- together with the request to sender or after some period?
- to which members? - fraction of admins? all admins?
- Member receiving XGrpRequestSkipped may not have requested messages, for example:
- request is for the older parent id, and member never received it himself (was not part of the group then or has gap in place of this message), or has gap between sent message parent and requested parent.
- member deleted parent(s), e.g. via periodic cleanup, or by deleting specific messages.
- don't fully delete group message records while in group? instead only overwrite content?
Message integrity is computed for received messages, can be updated on receiving requested message parents.
```haskell
data GroupMsgIntegrity
= GMIOk
| GMISkippedParents {skippedParents :: [MsgParentId]}
| GMIBadParentHash {knownParent :: MsgParentId, badParent :: MsgParentId} -- list?
```
```sql
CREATE TABLE message_integrity_records( -- message_hashes? group_messages?
message_integrity_record_id INTEGER PRIMARY KEY,
message_id INTEGER NOT NULL REFERENCES messages ON DELETE CASCADE, -- SET NULL?
group_id INTEGER NOT NULL REFERENCES groups ON DELETE CASCADE,
group_member_id INTEGER NOT NULL REFERENCES group_members ON DELETE CASCADE,
member_id BLOB NOT NULL,
member_msg_id INTEGER NOT NULL, -- shared_msg_id?
msg_hash BLOB NOT NULL,
msg_integrity TEXT NOT NULL, -- computed for received messages, for sent always Ok?
created_at TEXT NOT NULL DEFAULT(datetime('now')),
updated_at TEXT NOT NULL DEFAULT(datetime('now'))
);
-- many to many table for message_integrity_records table
-- (parent can have multiple children, child can have multiple parents)
-- parent can be null if it wasn't received
CREATE TABLE message_parents(
message_parent_id INTEGER PRIMARY KEY,
message_integrity_record_id INTEGER NOT NULL REFERENCES message_integrity_record_id ON DELETE CASCADE,
message_parent_integrity_record_id INTEGER REFERENCES message_integrity_record_id ON DELETE CASCADE,
msg_parent_member_id BLOB NOT NULL,
msg_parent_member_msg_id INTEGER NOT NULL,
msg_hash BLOB NOT NULL,
created_at TEXT NOT NULL DEFAULT(datetime('now')),
updated_at TEXT NOT NULL DEFAULT(datetime('now'))
);
```
How should message integrity errors be displayed in UI?
- Displaying skipped parent errors would clutter UI due to delays in delivery. Probably they shouldn't be displayed.
- Integrity violations (hashes not matching) should be displayed on respective chat items.
- if integrity is on AppMessage level for all chat events - not all messages have corresponding chat items, create internal chat items?
- if it's on the level of content messages, updates / etc. can be high above in message history, deletes can be not visible at all (full delete).
- how to get reference to message via chat item when loading chat items? Integrity violation can be on a message different than chat item's created_by_msg_id message. For each chat item load integrity of all messages via chat_item_messages?
- If integrity errors are only displayed on integrity violations, for malicious member to work around it and send different message to different group members could he specify unknown (far into future or past) message id, instead of incorrect one? Sender then wouldn't respond with skipped parents (and other members wouldn't be able to) - how to differentiate between this case and skipper parent error that is to be ignored in UI?
- Should it be prohibited to not send MsgIds (to avoid message integrity check) if member protocol version supports it? Should it be prohibited at all and group with integrity be separated? How to distinguish between messages sent without integrity fields and messages with skipped parents in UI?
- Not showing skipped parents integrity error in UI would lead user to believe integrity is preserved, and integrity violation can be revealed later. If conversation is time sensitive member may react to message considering it conversation integrity wasn't breached, and integrity violation may be revealed later. Having eventual integrity may not be better than having no integrity at all, and may even be worse because it produces false assumptions regarding conversation integrity. The goal can be narrowed to only restoring missed messages (gaps), without calculating integrity.
### Consistent group state
TODO
docs/rfcs/2023-10-20-group-integrity.md
+229
View File
@@ -0,0 +1,229 @@
# Group integrity
3 level of DAGs:
Owner
- group profile and permissions, admin invites and removals
- in case of gap vote before applying event
Admin
- member invites and removals
- prohibit to add and remove admins
- in case of gap most destructive wins
- link to owner dag
Messages
- in case of gap show history according to local graph, correct when owner or admin dag changes
- link to both admin and owner dags
```haskell
-- protocol
data MsgParent = MsgParent
{ memberId :: MemberId,
memberName :: String, -- recipient can use to display message if they don't have member introduced;
-- optional?
sharedMsgId :: SharedMsgId,
msgHash :: ByteString,
msgBody :: String? -- recipient can use to display message in case parent wasn't yet received;
-- sender can pack as many parents as fits into block
stored :: Bool -- whether sender has message stored, and it can be requested
}
data MsgIds = MsgIds -- include into chat event
{ sharedMsgId :: SharedMsgId,
ownerDAGMsgId :: SharedMsgId, -- list of parents?
adminDAGMsgId :: SharedMsgId,
parents :: [MsgParent]
}
-- model
data OwnerDAGEventParent
= ODEPKnown {eventId :: ?} -- DB id? sharedMsgId?
| ODEPUnknown {eventId :: ?}
data OwnerDAGEvent = DAGEvent
{ eventId :: ?,
parents :: [OwnerDAGEventParent]
}
data AdminDAGEventParent
= ADEPKnown {eventId :: ?}
| ADEPUnknown {eventId :: ?}
data AdminDAGEvent = DAGEvent
{ eventId :: ?,
ownerDAGEventId :: ?, -- [OwnerDAGEventParent] - parentIds? ?
parents :: [AdminDAGEventParent]
}
data MessagesDAGEventParent
= MDEPKnown {eventId :: ?}
| MDEPUnknown {eventId :: ?}
data MessagesDAGEvent = DAGEvent
{ eventId :: ?,
ownerDAGEventId :: ?, -- [OwnerDAGEventParent] - parentIds? ?
adminDAGEventId :: ?, -- [AdminDAGEventParent] - parentIds? ?
parents :: [MessagesDAGEventParent]
}
```
How to restore from destructive messages?
Even if all message parents are known, destructive logic of message should be applied after other members refer it.
How to workaround members maliciously referring non-existent parents?
For example, this can lead to an owner preventing group updates.
```
-- should dag be maintained in memory? older events to be removed
-- read on event?
-- how long into past to get dag?
ClassifiedEvent = OwnerEvent | AdminEvent | MsgEvent
def processEvent(e: Event) =
classifiedEvent <- classifyEvent(e)
case classifiedEvent of
OwnerEvent oe -> processOwnerEvent(oe)
AdminEvent ae -> processAdminEvent(ae)
MsgEvent me -> processMsgEvent(me)
def classifyEvent(e: Event) -> ClassifiedEvent? =
case e of
XMsgNew -> MsgEvent
XMsgFileDescr -> Nothing -- different per member
XMsgFileCancel -> MsgEvent
XMsgUpdate -> MsgEvent
XMsgDel -> MsgEvent
XMsgReact -> MsgEvent
XFile -> MsgEvent
XFileCancel -> MsgEvent
XFileAcptInv -> Nothing -- different per member
XGrpMemNew -> OwnerEvent -- sent by owner, new member is admin or owner
or AdminEvent -- sent by admin (or by owner and new member role is less than admin?)
-- problem: if member role changes, members can add event to different dags
-- what should define member role?
XGrpMemIntro -> Nothing -- received only by invitee
XGrpMemInv -> Nothing -- received only by host
XGrpMemFwd -> Nothing -- different per member; not received by invitee
XGrpMemRole -> OwnerEvent -- sent by owner about owner or admin
or AdminEvent -- sent by admin (or by owner about member with role less than admin?)
XGrpMemDel -> OwnerEvent -- sent by owner about owner or admin
or AdminEvent -- sent by admin (or by owner about member with role less than admin?)
XGrpLeave -> MsgEvent
XGrpDel -> OwnerEvent
XGrpInfo -> OwnerEvent
XGrpDirectInv -> Nothing -- received by single member
XInfoProbe -> Nothing -- per member
XInfoProbeCheck -> Nothing -- per member
XInfoProbeOk -> Nothing -- per member
BFileChunk -> Nothing -- could be MsgEvent?
_ -> Nothing -- not supported in groups
-- # owner events
def processOwnerEvent(oe: OwnerEvent) =
process every owner event after owners reach consensus
// def processOwnerEvent(oe: OwnerEvent) =
// addOwnerDagEvent(oe)
// applyOwnerDagEvent(oe)
//
// def addOwnerDagEvent(oe: OwnerEvent) =
// if (any parent of oe not in dag):
// buffer until all parents are in ownerDag
// else
// add oe to ownerDag
//
// def applyOwnerDagEvent(oe: OwnerEvent) =
// case oe of
// -- process XGrpMemNew, XGrpMemRole, XGrpMemDel same as for admin dag (see below), or should vote for all events?
// XGrpMemNew -> ...
// XGrpMemRole -> ...
// XGrpMemDel -> ...
// -- how to vote - to depend on action (group - manual, update - automatic?);
// -- wait for voting always, or if event has unknown parents? (gaps in dag)
// -- how to treat delayed integrity violation - owner sending message to select members
// XGrpDel ->
// -- create "pending group deletion", wait for confirmation from majority of owners?
// -- new protocol requiring user action from other owners?
// XGrpInfo ->
// -- create "unconfirmed group profile update", remember prev group profile
// -- remove from "unconfirmed group profile update" when this event is in dag and not a leaf?
// -- if another group profile update event is received, revert "unconfirmed" event, don't apply new
// -- so if more than one update is received while dag is not merged to single vertice, all updates are not applied
// -- - this would likely lock out owners from any future updates
// -- - merge to new starting point after some time passes?
// -- - mark parents that are never received and so always block graph merging as special type?
-- # admin events
def processAdminEvent(ae: AdminEvent) =
lookup in owner dag - does member still have permission?
addAdminDagEvent(ae)
applyAdminDagEvent(ae)
def addAdminDagEvent(ae: AdminEvent) =
if (any parent of ae not in dag):
buffer until all parents are in adminDag
else
add ae to adminDag
def applyAdminDagEvent(ae: AdminEvent) =
case ae of
XGrpMemNew ->
-- handles case where messages from 2 admins about member addition and deletion arrive out of order
if member is not in "unconfirmed member deletions":
add member
XGrpMemRole ->
add role change to "unconfirmed role change"
-- remove from "unconfirmed role change" when this event is in dag and not a leaf?
if another role change already in "unconfirmed role change":
if new role is less than role in "unconfirmed role change":
change role -- role change applies in direction of lower role
XGrpMemDel ->
add member to "unconfirmed member deletions"
-- remove from "unconfirmed member deletions" when this event is in dag and not a leaf?
if member found by memberId:
delete member
-- ^ problem: if later admin event turns out to fail integrity check, how to revert it?
-- member deletion: don't apply until in graph and not a leaf
-- role change: remember previous role and revert
-- member addition: delete member
-- # message events
def processMsgEvent(me: MsgEvent) =
lookup points in owner and admin dag?
- does member have permission to send event? (role changed/removed)
addMsgDagEvent(me)
applyMsgEvent(me)
def addMsgDagEvent(me: MsgEvent) =
for me.parents not in msgDag:
add MDEPUnknown parent to msgDag
add me to msgDag
def applyMsgEvent(me: MsgEvent) =
case me of
XMsgNew -> message to view
-- start process waiting for missing parents; if parents are not received:
-- can be shown as integrity violation if parents are not received
-- can be shown as integrity violation if other members don't refer it?
XMsgFileCancel -> cancel file immediately
-- wait for missing parents / referrals similarly to XMsgNew
-- restart file reception on integrity violation?
XMsgUpdate -> update to view -- same as XMsgNew
XMsgDel -> mark deleted, don't apply full delete until parents/referrals are received?
XMsgReact -> to view -- same as XMsgNew
XFile -> -- deprecate?
XFileCancel -> cancel -- same as XMsgFileCancel
XGrpLeave -> mark member as left, don't delete member connection immediately
-- member may try to maliciously remove connections selectively
-- wait for integrity check
```
# Admin blockchain
Suppose admin DAG is replaced with blockchain, with a conflict resolution protocol to provide consistency of membership changes. Take Simplex (not to confuse with SimpleX chat) protocol (https://simplex.blog/). To reach BFT consensus and make progress, 2n/3 votes on block proposals are required, and it's assumed `f < n/3` where f is number of malicious actors. In a highly asynchronous setting of decentralized groups operated by mobile devices, progress seems unlikely or very slow. Should "admin participation" be hosted?