check display name

cabal.project

@@ -12,7 +12,7 @@ constraints: zip +disable-bzip2 +disable-zstd
 source-repository-package
     type: git
     location: https://github.com/simplex-chat/simplexmq.git
-    tag: 3525e9d555aa7b207a8327a844161f55c19d5d2b
+    tag: caeeb2df9ccca29a6bb504886736502d081fba0e
 
 source-repository-package
     type: git

docs/rfcs/2024-02-12-database-migration.md

@@ -1,130 +0,0 @@
-# Database migration and other operations
-
-## Problem
-
-Migrating database to another device is very complex for most people - it is multi-step and error-prone.
-
-In addition to that, any database operation is confusing as it requires stopping chat.
-
-## Solution
-
-Let users migrate database to another device by scanning QR code.
-
-Simplify other database operations by removing the need to compose multiple actions, stop chat, etc.
-
-To support it, we already added the way to represent the file as link/QR code (by uploading file description to XFTP, and supporting "recursive" descriptions).
-
-There will be these actions in the Database settings (no stop/start chat toggle):
-
-- Export database.
-- Import database.
-- Migrate from another device.
-- Set passphrase (or Change passphrase if it was set).
-- Remove passphrase from device / Store passphrase on the device.
-
-Stop chat toggle will be moved to dev tools.
-
-Migrate to another device will be available in the top part of the settings, 
-
-
-### Database export
-
-Currently, it requires these steps:
-
-1. Open Database settings.
-2. Stop chat (many users don't understand it).
-3. Tap "Export database" in settings.
-4. Look at the alert that says "set passphrase".
-5. Tap Ok.
-6. Tap Set passphrase.
-7. Enter passphrase and confirm.
-8. Exit back to Database settings.
-9. Tap "Export database" again.
-10. Choose file location and save.
-11. Tap "New archive".
-12. Remove exported archive.
-
-These steps are all very confusing, and if they were to stay as composable steps, they belong to dev tools.
-
-Instead we can offer these simple steps:
-
-1. Open Database settings.
-2. Tap "Export database".
-3. Alert will appear saying: "The chat will stop, and you will need to set (or verify) database passphrase. Continue?".
-4. Tap "Ok".
-5. Enter passphrase and confirm in the window that appears (or verify if it was already set, possibly allowing to skip this step).
-7. Choose whether to save file or upload to XFTP and generate link.
-8. File: choose file location and save.
-   Link: show upload progress and then show link to copy.
-9. Alert will appear saying: "Database exported!", exported archive will be automatically removed.
-
-So instead of asking users to understand the required sequence of steps, we will guide them through the required process.
-
-### Database import
-
-1. Open Database settings.
-2. Tap "Import database".
-3. Alert will appear saying: "The chat will stop, you will import?".
-4. File: choose file location and tap "Import".
-   Link: paste link (or scan QR code) and tap "Import".
-5. Confirm to replace database.
-6. Start chat automatically once imported.
-
-### Set or change passphrase
-
-1. Open Database settings.
-2. Tap "Set passphrase" or "Change passphrase" (if it was set).
-3. Choose - store passphrase on the device or enter it every time the app starts.
-
-### Remove / store passphrase from the device
-
-To remove:
-
-1. Open Database settings.
-2. Tap "Remove passphrase".
-3. Confirm to remove passphrase in alert.
-4. Button is replaced with Store.
-
-To store:
-
-1. Open Database settings.
-2. Tap "Store passphrase".
-3. Enter current passphrase - it is verified.
-4. Button is replaced with Remove.
-
-### Migrate database to / from another device
-
-#### User experience
-
-This function is the most important, and it should be available from the main section in settings, under "Use from desktop" (or under "Link from mobile" on desktop).
-
-On the receiving device it will be available via Database settings and also on the Onboarding screen, so users don't need to create a profile.
-
-The steps are:
-
-On the source device:
-1. Tap "Migrate to another device".
-2. The chat will stop showing "Stopping chat" to the user.
-3. If passphrase was:
-   - not set: make user set it in a separate screen.
-   - set: make user verify it.
-5. Show the screen to confirm the upload.
-6. Upload progress (full screen circular progress showing the share, with the %s and total/uploaded size) will be shown.
-7. Once upload is completed, show QR code (with option to copy link), instruct to tap "Migrate from another device" on the receiving device.
-
-On the receiving device:
-2. Tap "Migrate from another device".
-2. The chat will stop (if not from Onboarding) showing "Stopping chat" to the user.
-4. Scan QR code (with option to paste link on desktop only).
-5. Show similar download progress, but probably in reversed direction - design TBC.
-6. Once download is completed, show "Replace the current database" (if not from Onboarding).
-7. Once imported, start chat automatically, and once chat started show "Tap remove database on source device".
-
-On the source device:
-1. Tap "Remove database" on the showing screen (this should also remove uploaded file).
-
-#### Implementation considerations
-
-The latest updates allow uploading and downloading XFTP files without messages.
-
-So to perform the above, the second instance of the chat controller will be required, that probably requires supporting additional/optional chat controller parameter in the APIs that are required for that process.

package.yaml

@@ -1,5 +1,5 @@
 name: simplex-chat
-version: 5.6.0.0
+version: 5.5.3.0
 #synopsis:
 #description:
 homepage: https://github.com/simplex-chat/simplex-chat#readme

simplex-chat.cabal

@@ -5,7 +5,7 @@ cabal-version: 1.12
 -- see: https://github.com/sol/hpack
 
 name:           simplex-chat
-version:        5.6.0.0
+version:        5.5.3.0
 category:       Web, System, Services, Cryptography
 homepage:       https://github.com/simplex-chat/simplex-chat#readme
 author:         simplex.chat
@@ -133,7 +133,6 @@ library
       Simplex.Chat.Migrations.M20240104_members_profile_update
       Simplex.Chat.Migrations.M20240115_block_member_for_all
       Simplex.Chat.Migrations.M20240122_indexes
-      Simplex.Chat.Migrations.M20240214_redirect_file_id
       Simplex.Chat.Mobile
       Simplex.Chat.Mobile.File
       Simplex.Chat.Mobile.Shared

src/Simplex/Chat.hs

@@ -6,7 +6,6 @@
 {-# LANGUAGE MultiWayIf #-}
 {-# LANGUAGE NamedFieldPuns #-}
 {-# LANGUAGE OverloadedStrings #-}
-{-# LANGUAGE PatternSynonyms #-}
 {-# LANGUAGE RankNTypes #-}
 {-# LANGUAGE ScopedTypeVariables #-}
 {-# LANGUAGE TupleSections #-}
@@ -82,8 +81,7 @@ import Simplex.Chat.Types.Util
 import Simplex.Chat.Util (encryptFile, shuffle)
 import Simplex.FileTransfer.Client.Main (maxFileSize)
 import Simplex.FileTransfer.Client.Presets (defaultXFTPServers)
-import Simplex.FileTransfer.Description (FileDescriptionURI (..), ValidFileDescription, gb, kb, mb)
+import Simplex.FileTransfer.Description (ValidFileDescription, gb, kb, mb)
-import qualified Simplex.FileTransfer.Description as FD
 import Simplex.FileTransfer.Protocol (FileParty (..), FilePartyI)
 import Simplex.Messaging.Agent as Agent
 import Simplex.Messaging.Agent.Client (AgentStatsKey (..), SubInfo (..), agentClientStore, getAgentWorkersDetails, getAgentWorkersSummary, temporaryAgentError)
@@ -772,18 +770,28 @@ processChatCommand' vr = \case
     CTContactConnection -> pure $ chatCmdError (Just user) "not supported"
     where
       xftpSndFileTransfer :: User -> CryptoFile -> Integer -> Int -> ContactOrGroup -> m (FileInvitation, CIFile 'MDSnd, FileTransferMeta)
-      xftpSndFileTransfer user file fileSize n contactOrGroup = do
+      xftpSndFileTransfer user file@(CryptoFile filePath cfArgs) fileSize n contactOrGroup = do
-        (fInv, ciFile, ft) <- xftpSndFileTransfer_ user file fileSize n $ Just contactOrGroup
+        let fileName = takeFileName filePath
+            fileDescr = FileDescr {fileDescrText = "", fileDescrPartNo = 0, fileDescrComplete = False}
+            fInv = xftpFileInvitation fileName fileSize fileDescr
+        fsFilePath <- toFSFilePath filePath
+        let srcFile = CryptoFile fsFilePath cfArgs
+        aFileId <- withAgent $ \a -> xftpSendFile a (aUserId user) srcFile (roundedFDCount n)
+        -- TODO CRSndFileStart event for XFTP
+        chSize <- asks $ fileChunkSize . config
+        ft@FileTransferMeta {fileId} <- withStore' $ \db -> createSndFileTransferXFTP db user contactOrGroup file fInv (AgentSndFileId aFileId) chSize
+        let fileSource = Just $ CryptoFile filePath cfArgs
+            ciFile = CIFile {fileId, fileName, fileSize, fileSource, fileStatus = CIFSSndStored, fileProtocol = FPXFTP}
         case contactOrGroup of
           CGContact Contact {activeConn} -> forM_ activeConn $ \conn ->
-            withStore' $ \db -> createSndFTDescrXFTP db user Nothing conn ft dummyFileDescr
+            withStore' $ \db -> createSndFTDescrXFTP db user Nothing conn ft fileDescr
           CGGroup (Group _ ms) -> forM_ ms $ \m -> saveMemberFD m `catchChatError` (toView . CRChatError (Just user))
             where
               -- we are not sending files to pending members, same as with inline files
               saveMemberFD m@GroupMember {activeConn = Just conn@Connection {connStatus}} =
                 when ((connStatus == ConnReady || connStatus == ConnSndReady) && not (connDisabled conn)) $
                   withStore' $
-                    \db -> createSndFTDescrXFTP db user (Just m) conn ft dummyFileDescr
+                    \db -> createSndFTDescrXFTP db user (Just m) conn ft fileDescr
               saveMemberFD _ = pure ()
         pure (fInv, ciFile, ft)
       unzipMaybe3 :: Maybe (a, b, c) -> (Maybe a, Maybe b, Maybe c)
@@ -1951,16 +1959,16 @@ processChatCommand' vr = \case
           | otherwise -> do
               fileAgentConnIds <- cancelSndFile user ftm fts True
               deleteAgentConnectionsAsync user fileAgentConnIds
-              withStore (\db -> liftIO $ lookupChatRefByFileId db user fileId) >>= \case
+              sharedMsgId <- withStore $ \db -> getSharedMsgIdByFileId db userId fileId
-                Nothing -> pure ()
+              withStore (\db -> getChatRefByFileId db user fileId) >>= \case
-                Just (ChatRef CTDirect contactId) -> do
+                ChatRef CTDirect contactId -> do
-                  (contact, sharedMsgId) <- withStore $ \db -> (,) <$> getContact db user contactId <*> getSharedMsgIdByFileId db userId fileId
+                  contact <- withStore $ \db -> getContact db user contactId
                   void . sendDirectContactMessage contact $ XFileCancel sharedMsgId
-                Just (ChatRef CTGroup groupId) -> do
+                ChatRef CTGroup groupId -> do
-                  (Group gInfo ms, sharedMsgId) <- withStore $ \db -> (,) <$> getGroup db vr user groupId <*> getSharedMsgIdByFileId db userId fileId
+                  Group gInfo ms <- withStore $ \db -> getGroup db vr user groupId
                   void . sendGroupMessage user gInfo ms $ XFileCancel sharedMsgId
-                Just _ -> throwChatError $ CEFileInternal "invalid chat ref for file transfer"
+                _ -> throwChatError $ CEFileInternal "invalid chat ref for file transfer"
-              ci <- withStore $ \db -> lookupChatItemByFileId db vr user fileId
+              ci <- withStore $ \db -> getChatItemByFileId db vr user fileId
               pure $ CRSndFileCancelled user ci ftm fts
           where
             fileCancelledOrCompleteSMP SndFileTransfer {fileStatus = s} =
@@ -1971,7 +1979,7 @@ processChatCommand' vr = \case
           | otherwise -> case xftpRcvFile of
               Nothing -> do
                 cancelRcvFileTransfer user ftr >>= mapM_ (deleteAgentConnectionAsync user)
-                ci <- withStore $ \db -> lookupChatItemByFileId db vr user fileId
+                ci <- withStore $ \db -> getChatItemByFileId db vr user fileId
                 pure $ CRRcvFileCancelled user ci ftr
               Just XFTPRcvFile {agentRcvFileId} -> do
                 forM_ (liveRcvFileTransferPath ftr) $ \filePath -> do
@@ -1984,21 +1992,18 @@ processChatCommand' vr = \case
                     updateCIFileStatus db user fileId CIFSRcvInvitation
                     updateRcvFileStatus db fileId FSNew
                     updateRcvFileAgentId db fileId Nothing
-                  lookupChatItemByFileId db vr user fileId
+                  getChatItemByFileId db vr user fileId
                 pure $ CRRcvFileCancelled user ci ftr
   FileStatus fileId -> withUser $ \user -> do
-    withStore (\db -> lookupChatItemByFileId db vr user fileId) >>= \case
+    ci@(AChatItem _ _ _ ChatItem {file}) <- withStore $ \db -> getChatItemByFileId db vr user fileId
-      Nothing -> do
+    case file of
+      Just CIFile {fileProtocol = FPLocal} ->
+        throwChatError $ CECommandError "not supported for local files"
+      Just CIFile {fileProtocol = FPXFTP} ->
+        pure $ CRFileTransferStatusXFTP user ci
+      _ -> do
         fileStatus <- withStore $ \db -> getFileTransferProgress db user fileId
         pure $ CRFileTransferStatus user fileStatus
-      Just ci@(AChatItem _ _ _ ChatItem {file}) -> case file of
-        Just CIFile {fileProtocol = FPLocal} ->
-          throwChatError $ CECommandError "not supported for local files"
-        Just CIFile {fileProtocol = FPXFTP} ->
-          pure $ CRFileTransferStatusXFTP user ci
-        _ -> do
-          fileStatus <- withStore $ \db -> getFileTransferProgress db user fileId
-          pure $ CRFileTransferStatus user fileStatus
   ShowProfile -> withUser $ \user@User {profile} -> pure $ CRUserProfile user (fromLocalProfile profile)
   UpdateProfile displayName fullName -> withUser $ \user@User {profile} -> do
     let p = (fromLocalProfile profile :: Profile) {displayName = displayName, fullName = fullName}
@@ -2053,13 +2058,6 @@ processChatCommand' vr = \case
   StopRemoteCtrl -> withUser_ $ stopRemoteCtrl >> ok_
   ListRemoteCtrls -> withUser_ $ CRRemoteCtrlList <$> listRemoteCtrls
   DeleteRemoteCtrl rc -> withUser_ $ deleteRemoteCtrl rc >> ok_
-  APIUploadStandaloneFile userId file -> withUserId userId $ \user -> do
-    fileSize <- liftIO $ CF.getFileContentsSize file
-    (_, _, fileTransferMeta) <- xftpSndFileTransfer_ user file fileSize 1 Nothing
-    pure CRSndStandaloneFileCreated {user, fileTransferMeta}
-  APIDownloadStandaloneFile userId uri file -> withUserId userId $ \user -> do
-    ft <- receiveViaURI user uri file
-    pure $ CRRcvStandaloneFileCreated user ft
   QuitChat -> liftIO exitSuccess
   ShowVersion -> do
     -- simplexmqCommitQ makes iOS builds crash m(
@@ -2813,19 +2811,6 @@ receiveViaCompleteFD user fileId RcvFileDescr {fileDescrText, fileDescrComplete}
     startReceivingFile user fileId
     withStoreCtx' (Just "receiveViaCompleteFD, updateRcvFileAgentId") $ \db -> updateRcvFileAgentId db fileId (Just $ AgentRcvFileId aFileId)
 
-receiveViaURI :: ChatMonad m => User -> FileDescriptionURI -> CryptoFile -> m RcvFileTransfer
-receiveViaURI user@User {userId} FileDescriptionURI {description} cf@CryptoFile {cryptoArgs} = do
-  fileId <- withStore $ \db -> createRcvStandaloneFileTransfer db userId cf fileSize chunkSize
-  aFileId <- withAgent $ \a -> xftpReceiveFile a (aUserId user) description cryptoArgs
-  withStore $ \db -> do
-    liftIO $ do
-      updateRcvFileStatus db fileId FSConnected
-      updateCIFileStatus db user fileId $ CIFSRcvTransfer 0 1
-      updateRcvFileAgentId db fileId (Just $ AgentRcvFileId aFileId)
-    getRcvFileTransfer db user fileId
-  where
-    FD.ValidFileDescription FD.FileDescription {size = FD.FileSize fileSize, chunkSize = FD.FileSize chunkSize} = description
-
 startReceivingFile :: ChatMonad m => User -> FileTransferId -> m ()
 startReceivingFile user fileId = do
   vr <- chatVersionRange
@@ -3291,7 +3276,7 @@ processAgentMsgSndFile _corrId aFileId msg =
   where
     process :: User -> m ()
     process user = do
-      (ft@FileTransferMeta {fileId, xftpRedirectFor, cancelled}, sfts) <- withStore $ \db -> do
+      (ft@FileTransferMeta {fileId, cancelled}, sfts) <- withStore $ \db -> do
         fileId <- getXFTPSndFileDBId db user $ AgentSndFileId aFileId
         getSndFileTransfer db user fileId
       vr <- chatVersionRange
@@ -3300,76 +3285,61 @@ processAgentMsgSndFile _corrId aFileId msg =
           let status = CIFSSndTransfer {sndProgress, sndTotal}
           ci <- withStore $ \db -> do
             liftIO $ updateCIFileStatus db user fileId status
-            lookupChatItemByFileId db vr user fileId
+            getChatItemByFileId db vr user fileId
           toView $ CRSndFileProgressXFTP user ci ft sndProgress sndTotal
         SFDONE sndDescr rfds -> do
           withStore' $ \db -> setSndFTPrivateSndDescr db user fileId (fileDescrText sndDescr)
-          ci <- withStore $ \db -> lookupChatItemByFileId db vr user fileId
+          ci@(AChatItem _ d cInfo _ci@ChatItem {meta = CIMeta {itemSharedMsgId = msgId_, itemDeleted}}) <-
-          case ci of
+            withStore $ \db -> getChatItemByFileId db vr user fileId
-            Nothing -> do
+          case (msgId_, itemDeleted) of
-              withAgent (`xftpDeleteSndFileInternal` aFileId)
+            (Just sharedMsgId, Nothing) -> do
-              withStore' $ \db -> createExtraSndFTDescrs db user fileId (map fileDescrText rfds)
+              when (length rfds < length sfts) $ throwChatError $ CEInternalError "not enough XFTP file descriptions to send"
-              case mapMaybe fileDescrURI rfds of
+              -- TODO either update database status or move to SFPROG
-                [] -> case rfds of
+              toView $ CRSndFileProgressXFTP user ci ft 1 1
-                  [] -> logError "File sent without receiver descriptions" -- should not happen
+              case (rfds, sfts, d, cInfo) of
-                  (rfd : _) -> xftpSndFileRedirect user fileId rfd >>= toView . CRSndFileRedirectStartXFTP user ft
+                (rfd : extraRFDs, sft : _, SMDSnd, DirectChat ct) -> do
-                uris -> do
+                  withStore' $ \db -> createExtraSndFTDescrs db user fileId (map fileDescrText extraRFDs)
-                  ft' <- maybe (pure ft) (\fId -> withStore $ \db -> getFileTransferMeta db user fId) xftpRedirectFor
+                  msgDeliveryId <- sendFileDescription sft rfd sharedMsgId $ sendDirectContactMessage ct
-                  toView $ CRSndStandaloneFileComplete user ft' uris
+                  withStore' $ \db -> updateSndFTDeliveryXFTP db sft msgDeliveryId
-            Just (AChatItem _ d cInfo _ci@ChatItem {meta = CIMeta {itemSharedMsgId = msgId_, itemDeleted}}) ->
+                  withAgent (`xftpDeleteSndFileInternal` aFileId)
-              case (msgId_, itemDeleted) of
+                (_, _, SMDSnd, GroupChat g@GroupInfo {groupId}) -> do
-                (Just sharedMsgId, Nothing) -> do
+                  ms <- withStore' $ \db -> getGroupMembers db user g
-                  when (length rfds < length sfts) $ throwChatError $ CEInternalError "not enough XFTP file descriptions to send"
+                  let rfdsMemberFTs = zip rfds $ memberFTs ms
-                  -- TODO either update database status or move to SFPROG
+                      extraRFDs = drop (length rfdsMemberFTs) rfds
-                  toView $ CRSndFileProgressXFTP user ci ft 1 1
+                  withStore' $ \db -> createExtraSndFTDescrs db user fileId (map fileDescrText extraRFDs)
-                  case (rfds, sfts, d, cInfo) of
+                  forM_ rfdsMemberFTs $ \mt -> sendToMember mt `catchChatError` (toView . CRChatError (Just user))
-                    (rfd : extraRFDs, sft : _, SMDSnd, DirectChat ct) -> do
+                  ci' <- withStore $ \db -> do
-                      withStore' $ \db -> createExtraSndFTDescrs db user fileId (map fileDescrText extraRFDs)
+                    liftIO $ updateCIFileStatus db user fileId CIFSSndComplete
-                      msgDeliveryId <- sendFileDescription sft rfd sharedMsgId $ sendDirectContactMessage ct
+                    getChatItemByFileId db vr user fileId
-                      withStore' $ \db -> updateSndFTDeliveryXFTP db sft msgDeliveryId
+                  withAgent (`xftpDeleteSndFileInternal` aFileId)
-                      withAgent (`xftpDeleteSndFileInternal` aFileId)
+                  toView $ CRSndFileCompleteXFTP user ci' ft
-                    (_, _, SMDSnd, GroupChat g@GroupInfo {groupId}) -> do
+                  where
-                      ms <- withStore' $ \db -> getGroupMembers db user g
+                    memberFTs :: [GroupMember] -> [(Connection, SndFileTransfer)]
-                      let rfdsMemberFTs = zip rfds $ memberFTs ms
+                    memberFTs ms = M.elems $ M.intersectionWith (,) (M.fromList mConns') (M.fromList sfts')
-                          extraRFDs = drop (length rfdsMemberFTs) rfds
-                      withStore' $ \db -> createExtraSndFTDescrs db user fileId (map fileDescrText extraRFDs)
-                      forM_ rfdsMemberFTs $ \mt -> sendToMember mt `catchChatError` (toView . CRChatError (Just user))
-                      ci' <- withStore $ \db -> do
-                        liftIO $ updateCIFileStatus db user fileId CIFSSndComplete
-                        getChatItemByFileId db vr user fileId
-                      withAgent (`xftpDeleteSndFileInternal` aFileId)
-                      toView $ CRSndFileCompleteXFTP user ci' ft
                       where
-                        memberFTs :: [GroupMember] -> [(Connection, SndFileTransfer)]
+                        mConns' = mapMaybe useMember ms
-                        memberFTs ms = M.elems $ M.intersectionWith (,) (M.fromList mConns') (M.fromList sfts')
+                        sfts' = mapMaybe (\sft@SndFileTransfer {groupMemberId} -> (,sft) <$> groupMemberId) sfts
-                          where
+                        useMember GroupMember {groupMemberId, activeConn = Just conn@Connection {connStatus}}
-                            mConns' = mapMaybe useMember ms
+                          | (connStatus == ConnReady || connStatus == ConnSndReady) && not (connDisabled conn) = Just (groupMemberId, conn)
-                            sfts' = mapMaybe (\sft@SndFileTransfer {groupMemberId} -> (,sft) <$> groupMemberId) sfts
+                          | otherwise = Nothing
-                            useMember GroupMember {groupMemberId, activeConn = Just conn@Connection {connStatus}}
+                        useMember _ = Nothing
-                              | (connStatus == ConnReady || connStatus == ConnSndReady) && not (connDisabled conn) = Just (groupMemberId, conn)
+                    sendToMember :: (ValidFileDescription 'FRecipient, (Connection, SndFileTransfer)) -> m ()
-                              | otherwise = Nothing
+                    sendToMember (rfd, (conn, sft)) =
-                            useMember _ = Nothing
+                      void $ sendFileDescription sft rfd sharedMsgId $ \msg' -> sendDirectMessage conn msg' $ GroupId groupId
-                        sendToMember :: (ValidFileDescription 'FRecipient, (Connection, SndFileTransfer)) -> m ()
+                _ -> pure ()
-                        sendToMember (rfd, (conn, sft)) =
+            _ -> pure () -- TODO error?
-                          void $ sendFileDescription sft rfd sharedMsgId $ \msg' -> sendDirectMessage conn msg' $ GroupId groupId
-                    _ -> pure ()
-                _ -> pure () -- TODO error?
         SFERR e
           | temporaryAgentError e ->
               throwChatError $ CEXFTPSndFile fileId (AgentSndFileId aFileId) e
           | otherwise -> do
               ci <- withStore $ \db -> do
                 liftIO $ updateFileCancelled db user fileId CIFSSndError
-                lookupChatItemByFileId db vr user fileId
+                getChatItemByFileId db vr user fileId
               withAgent (`xftpDeleteSndFileInternal` aFileId)
-              toView $ CRSndFileError user ci ft
+              toView $ CRSndFileError user ci
       where
         fileDescrText :: FilePartyI p => ValidFileDescription p -> T.Text
         fileDescrText = safeDecodeUtf8 . strEncode
-        fileDescrURI :: ValidFileDescription 'FRecipient -> Maybe T.Text
-        fileDescrURI vfd = if T.length uri < FD.qrSizeLimit then Just uri else Nothing
-          where
-            uri = decodeLatin1 . strEncode $ FD.fileDescriptionURI vfd
         sendFileDescription :: SndFileTransfer -> ValidFileDescription 'FRecipient -> SharedMsgId -> (ChatMsgEvent 'Json -> m (SndMessage, Int64)) -> m Int64
         sendFileDescription sft rfd msgId sendMsg = do
           let rfdText = fileDescrText rfd
@@ -3417,30 +3387,30 @@ processAgentMsgRcvFile _corrId aFileId msg =
           let status = CIFSRcvTransfer {rcvProgress, rcvTotal}
           ci <- withStore $ \db -> do
             liftIO $ updateCIFileStatus db user fileId status
-            lookupChatItemByFileId db vr user fileId
+            getChatItemByFileId db vr user fileId
-          toView $ CRRcvFileProgressXFTP user ci rcvProgress rcvTotal ft
+          toView $ CRRcvFileProgressXFTP user ci rcvProgress rcvTotal
         RFDONE xftpPath ->
           case liveRcvFileTransferPath ft of
             Nothing -> throwChatError $ CEInternalError "no target path for received XFTP file"
             Just targetPath -> do
               fsTargetPath <- toFSFilePath targetPath
               renameFile xftpPath fsTargetPath
-              ci_ <- withStore $ \db -> do
+              ci <- withStore $ \db -> do
                 liftIO $ do
                   updateRcvFileStatus db fileId FSComplete
                   updateCIFileStatus db user fileId CIFSRcvComplete
-                lookupChatItemByFileId db vr user fileId
+                getChatItemByFileId db vr user fileId
               agentXFTPDeleteRcvFile aFileId fileId
-              toView $ maybe (CRRcvStandaloneFileComplete user fsTargetPath ft) (CRRcvFileComplete user) ci_
+              toView $ CRRcvFileComplete user ci
         RFERR e
           | temporaryAgentError e ->
               throwChatError $ CEXFTPRcvFile fileId (AgentRcvFileId aFileId) e
           | otherwise -> do
               ci <- withStore $ \db -> do
                 liftIO $ updateFileCancelled db user fileId CIFSRcvError
-                lookupChatItemByFileId db vr user fileId
+                getChatItemByFileId db vr user fileId
               agentXFTPDeleteRcvFile aFileId fileId
-              toView $ CRRcvFileError user ci e ft
+              toView $ CRRcvFileError user ci e
 
 processAgentMessageConn :: forall m. ChatMonad m => VersionRange -> User -> ACorrId -> ConnId -> ACommand 'Agent 'AEConn -> m ()
 processAgentMessageConn vr user@User {userId} corrId agentConnId agentMessage = do
@@ -3648,15 +3618,18 @@ processAgentMessageConn vr user@User {userId} corrId agentConnId agentMessage =
             processErr cryptoErr = do
               let e@(mde, n) = agentMsgDecryptError cryptoErr
               ci_ <- withStore $ \db ->
-                getDirectChatItemLast db user contactId
+                getDirectChatItemsLast db user contactId 1 ""
                   >>= liftIO
                     . mapM (\(ci, content') -> updateDirectChatItem' db user contactId ci content' False Nothing)
-                    . mdeUpdatedCI e
+                    . (mdeUpdatedCI e <=< headMaybe)
               case ci_ of
                 Just ci -> toView $ CRChatItemUpdated user (AChatItem SCTDirect SMDRcv (DirectChat ct) ci)
                 _ -> do
                   toView $ CRContactRatchetSync user ct (RatchetSyncProgress rss cStats)
                   createInternalChatItem user (CDDirectRcv ct) (CIRcvDecryptionError mde n) Nothing
+            headMaybe = \case
+              x : _ -> Just x
+              _ -> Nothing
             ratchetSyncEventItem ct' = do
               toView $ CRContactRatchetSync user ct' (RatchetSyncProgress rss cStats)
               createInternalChatItem user (CDDirectRcv ct') (CIRcvConnEvent $ RCERatchetSync rss) Nothing
@@ -4116,10 +4089,10 @@ processAgentMessageConn vr user@User {userId} corrId agentConnId agentMessage =
           case err of
             SMP SMP.AUTH -> unless (fileStatus == FSCancelled) $ do
               ci <- withStore $ \db -> do
-                liftIO (lookupChatRefByFileId db user fileId) >>= \case
+                getChatRefByFileId db user fileId >>= \case
-                  Just (ChatRef CTDirect _) -> liftIO $ updateFileCancelled db user fileId CIFSSndCancelled
+                  ChatRef CTDirect _ -> liftIO $ updateFileCancelled db user fileId CIFSSndCancelled
                   _ -> pure ()
-                lookupChatItemByFileId db vr user fileId
+                getChatItemByFileId db vr user fileId
               toView $ CRSndFileRcvCancelled user ci ft
             _ -> throwChatError $ CEFileSend fileId err
         MSG meta _ _ -> withAckMessage' agentConnId conn meta $ pure ()
@@ -6301,19 +6274,12 @@ agentXFTPDeleteRcvFile aFileId fileId = do
   withStore' $ \db -> setRcvFTAgentDeleted db fileId
 
 agentXFTPDeleteSndFileRemote :: ChatMonad m => User -> XFTPSndFile -> FileTransferId -> m ()
-agentXFTPDeleteSndFileRemote user sndFile fileId = do
+agentXFTPDeleteSndFileRemote user XFTPSndFile {agentSndFileId = AgentSndFileId aFileId, privateSndFileDescr, agentSndFileDeleted} fileId =
-    -- the agent doesn't know about redirect, delete explicitly
+  unless agentSndFileDeleted $
-  redirect_ <- withStore' $ \db -> lookupFileTransferRedirectMeta db user fileId
+    forM_ privateSndFileDescr $ \sfdText -> do
-  forM_ redirect_ $ \FileTransferMeta {fileId = fileIdRedirect, xftpSndFile = sndFileRedirect_} ->
+      sd <- parseFileDescription sfdText
-    mapM_ (handleError (const $ pure ()) . remove fileIdRedirect) sndFileRedirect_
+      withAgent $ \a -> xftpDeleteSndFileRemote a (aUserId user) aFileId sd
-  remove fileId sndFile
+      withStore' $ \db -> setSndFTAgentDeleted db user fileId
-  where
-    remove fId XFTPSndFile {agentSndFileId = AgentSndFileId aFileId, privateSndFileDescr, agentSndFileDeleted} =
-      unless agentSndFileDeleted $ do
-        forM_ privateSndFileDescr $ \sfdText -> do
-          sd <- parseFileDescription sfdText
-          withAgent $ \a -> xftpDeleteSndFileRemote a (aUserId user) aFileId sd
-          withStore' $ \db -> setSndFTAgentDeleted db user fId
 
 userProfileToSend :: User -> Maybe Profile -> Maybe Contact -> Bool -> Profile
 userProfileToSend user@User {profile = p} incognitoProfile ct inGroup = do
@@ -6784,8 +6750,6 @@ chatCommandP =
       "/list remote ctrls" $> ListRemoteCtrls,
       "/stop remote ctrl" $> StopRemoteCtrl,
       "/delete remote ctrl " *> (DeleteRemoteCtrl <$> A.decimal),
-      "/_upload " *> (APIUploadStandaloneFile <$> A.decimal <* A.space <*> cryptoFileP),
-      "/_download " *> (APIDownloadStandaloneFile <$> A.decimal <* A.space <*> strP_ <*> cryptoFileP),
       ("/quit" <|> "/q" <|> "/exit") $> QuitChat,
       ("/version" <|> "/v") $> ShowVersion,
       "/debug locks" $> DebugLocks,
@@ -6973,29 +6937,3 @@ mkValidName = reverse . dropWhile isSpace . fst3 . foldl' addChar ("", '\NUL', 0
           | isPunctuation prev = validFirstChar || isSpace c || (punct < 3 && isPunctuation c)
           | otherwise = validFirstChar || isSpace c || isMark c || isPunctuation c
         validFirstChar = isLetter c || isNumber c || isSymbol c
-
-xftpSndFileTransfer_ :: ChatMonad m => User -> CryptoFile -> Integer -> Int -> Maybe ContactOrGroup -> m (FileInvitation, CIFile 'MDSnd, FileTransferMeta)
-xftpSndFileTransfer_ user file@(CryptoFile filePath cfArgs) fileSize n contactOrGroup_ = do
-  let fileName = takeFileName filePath
-      fInv = xftpFileInvitation fileName fileSize dummyFileDescr
-  fsFilePath <- toFSFilePath filePath
-  let srcFile = CryptoFile fsFilePath cfArgs
-  aFileId <- withAgent $ \a -> xftpSendFile a (aUserId user) srcFile (roundedFDCount n)
-  -- TODO CRSndFileStart event for XFTP
-  chSize <- asks $ fileChunkSize . config
-  ft@FileTransferMeta {fileId} <- withStore' $ \db -> createSndFileTransferXFTP db user contactOrGroup_ file fInv (AgentSndFileId aFileId) Nothing chSize
-  let fileSource = Just $ CryptoFile filePath cfArgs
-      ciFile = CIFile {fileId, fileName, fileSize, fileSource, fileStatus = CIFSSndStored, fileProtocol = FPXFTP}
-  pure (fInv, ciFile, ft)
-
-xftpSndFileRedirect :: ChatMonad m => User -> FileTransferId -> ValidFileDescription 'FRecipient -> m FileTransferMeta
-xftpSndFileRedirect user ftId vfd = do
-  let fileName = "redirect.yaml"
-      file = CryptoFile fileName Nothing
-      fInv = xftpFileInvitation fileName (fromIntegral $ B.length $ strEncode vfd) dummyFileDescr
-  aFileId <- withAgent $ \a -> xftpSendDescription a (aUserId user) vfd (roundedFDCount 1)
-  chSize <- asks $ fileChunkSize . config
-  withStore' $ \db -> createSndFileTransferXFTP db user Nothing file fInv (AgentSndFileId aFileId) (Just ftId) chSize
-
-dummyFileDescr :: FileDescr
-dummyFileDescr = FileDescr {fileDescrText = "", fileDescrPartNo = 0, fileDescrComplete = False}

src/Simplex/Chat/Controller.hs

@@ -59,7 +59,6 @@ import Simplex.Chat.Remote.Types
 import Simplex.Chat.Store (AutoAccept, StoreError (..), UserContactLink, UserMsgReceiptSettings)
 import Simplex.Chat.Types
 import Simplex.Chat.Types.Preferences
-import Simplex.FileTransfer.Description (FileDescriptionURI)
 import Simplex.Messaging.Agent (AgentClient, SubscriptionsInfo)
 import Simplex.Messaging.Agent.Client (AgentLocks, AgentWorkersDetails (..), AgentWorkersSummary (..), ProtocolTestFailure)
 import Simplex.Messaging.Agent.Env.SQLite (AgentConfig, NetworkConfig)
@@ -454,8 +453,6 @@ data ChatCommand
   | ListRemoteCtrls
   | StopRemoteCtrl -- Stop listening for announcements or terminate an active session
   | DeleteRemoteCtrl RemoteCtrlId -- Remove all local data associated with a remote controller session
-  | APIUploadStandaloneFile UserId CryptoFile
-  | APIDownloadStandaloneFile UserId FileDescriptionURI CryptoFile
   | QuitChat
   | ShowVersion
   | DebugLocks
@@ -596,26 +593,21 @@ data ChatResponse
   | CRRcvFileAccepted {user :: User, chatItem :: AChatItem}
   | CRRcvFileAcceptedSndCancelled {user :: User, rcvFileTransfer :: RcvFileTransfer}
   | CRRcvFileDescrNotReady {user :: User, chatItem :: AChatItem}
-  | CRRcvStandaloneFileCreated {user :: User, rcvFileTransfer :: RcvFileTransfer} -- returned by _download
+  | CRRcvFileStart {user :: User, chatItem :: AChatItem}
-  | CRRcvFileStart {user :: User, chatItem :: AChatItem} -- sent by chats
+  | CRRcvFileProgressXFTP {user :: User, chatItem :: AChatItem, receivedSize :: Int64, totalSize :: Int64}
-  | CRRcvFileProgressXFTP {user :: User, chatItem_ :: Maybe AChatItem, receivedSize :: Int64, totalSize :: Int64, rcvFileTransfer :: RcvFileTransfer}
   | CRRcvFileComplete {user :: User, chatItem :: AChatItem}
-  | CRRcvStandaloneFileComplete {user :: User, targetPath :: FilePath, rcvFileTransfer :: RcvFileTransfer}
+  | CRRcvFileCancelled {user :: User, chatItem :: AChatItem, rcvFileTransfer :: RcvFileTransfer}
-  | CRRcvFileCancelled {user :: User, chatItem_ :: Maybe AChatItem, rcvFileTransfer :: RcvFileTransfer}
   | CRRcvFileSndCancelled {user :: User, chatItem :: AChatItem, rcvFileTransfer :: RcvFileTransfer}
-  | CRRcvFileError {user :: User, chatItem_ :: Maybe AChatItem, agentError :: AgentErrorType, rcvFileTransfer :: RcvFileTransfer}
+  | CRRcvFileError {user :: User, chatItem :: AChatItem, agentError :: AgentErrorType}
   | CRSndFileStart {user :: User, chatItem :: AChatItem, sndFileTransfer :: SndFileTransfer}
   | CRSndFileComplete {user :: User, chatItem :: AChatItem, sndFileTransfer :: SndFileTransfer}
-  | CRSndFileRcvCancelled {user :: User, chatItem_ :: Maybe AChatItem, sndFileTransfer :: SndFileTransfer}
+  | CRSndFileRcvCancelled {user :: User, chatItem :: AChatItem, sndFileTransfer :: SndFileTransfer}
-  | CRSndFileCancelled {user :: User, chatItem_ :: Maybe AChatItem, fileTransferMeta :: FileTransferMeta, sndFileTransfers :: [SndFileTransfer]}
+  | CRSndFileCancelled {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta, sndFileTransfers :: [SndFileTransfer]}
-  | CRSndStandaloneFileCreated {user :: User, fileTransferMeta :: FileTransferMeta} -- returned by _upload
+  | CRSndFileStartXFTP {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta}
-  | CRSndFileStartXFTP {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta} -- not used
+  | CRSndFileProgressXFTP {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta, sentSize :: Int64, totalSize :: Int64}
-  | CRSndFileProgressXFTP {user :: User, chatItem_ :: Maybe AChatItem, fileTransferMeta :: FileTransferMeta, sentSize :: Int64, totalSize :: Int64}
-  | CRSndFileRedirectStartXFTP {user :: User, fileTransferMeta :: FileTransferMeta, redirectMeta :: FileTransferMeta}
   | CRSndFileCompleteXFTP {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta}
-  | CRSndStandaloneFileComplete {user :: User, fileTransferMeta :: FileTransferMeta, rcvURIs :: [Text]}
+  | CRSndFileCancelledXFTP {user :: User, chatItem :: AChatItem, fileTransferMeta :: FileTransferMeta}
-  | CRSndFileCancelledXFTP {user :: User, chatItem_ :: Maybe AChatItem, fileTransferMeta :: FileTransferMeta}
+  | CRSndFileError {user :: User, chatItem :: AChatItem}
-  | CRSndFileError {user :: User, chatItem_ :: Maybe AChatItem, fileTransferMeta :: FileTransferMeta}
   | CRUserProfileUpdated {user :: User, fromProfile :: Profile, toProfile :: Profile, updateSummary :: UserProfileUpdateSummary}
   | CRUserProfileImage {user :: User, profile :: Profile}
   | CRContactAliasUpdated {user :: User, toContact :: Contact}

src/Simplex/Chat/Messages.hs

@@ -360,24 +360,6 @@ mkCIMeta itemId itemContent itemText itemStatus itemSharedMsgId itemDeleted item
         _ -> False
    in CIMeta {itemId, itemTs, itemText, itemStatus, itemSharedMsgId, itemDeleted, itemEdited, itemTimed, itemLive, editable, forwardedByMember, createdAt, updatedAt}
 
-dummyMeta :: ChatItemId -> UTCTime -> Text -> CIMeta c 'MDSnd
-dummyMeta itemId ts itemText =
-  CIMeta
-    { itemId,
-      itemTs = ts,
-      itemText,
-      itemStatus = CISSndNew,
-      itemSharedMsgId = Nothing,
-      itemDeleted = Nothing,
-      itemEdited = False,
-      itemTimed = Nothing,
-      itemLive = Nothing,
-      editable = False,
-      forwardedByMember = Nothing,
-      createdAt = ts,
-      updatedAt = ts
-    }
-
 data CITimed = CITimed
   { ttl :: Int, -- seconds
     deleteAt :: Maybe UTCTime -- this is initially Nothing for received items, the timer starts when they are read

src/Simplex/Chat/Messages/CIContent.hs

@@ -139,7 +139,7 @@ data CIContent (d :: MsgDirection) where
   CISndModerated :: CIContent 'MDSnd
   CIRcvModerated :: CIContent 'MDRcv
   CIRcvBlocked :: CIContent 'MDRcv
-  CIInvalidJSON :: Text -> CIContent d -- this is also used for logical database errors, e.g. SEBadChatItem
+  CIInvalidJSON :: Text -> CIContent d
 -- ^ This type is used both in API and in DB, so we use different JSON encodings for the database and for the API
 -- ! ^ Nested sum types also have to use different encodings for database and API
 -- ! ^ to avoid breaking cross-platform compatibility, see RcvGroupEvent and SndGroupEvent

src/Simplex/Chat/Migrations/M20240214_redirect_file_id.hs

@@ -1,22 +0,0 @@
-{-# LANGUAGE QuasiQuotes #-}
-
-module Simplex.Chat.Migrations.M20240214_redirect_file_id where
-
-import Database.SQLite.Simple (Query)
-import Database.SQLite.Simple.QQ (sql)
-
-m20240214_redirect_file_id :: Query
-m20240214_redirect_file_id =
-  [sql|
-ALTER TABLE files ADD COLUMN redirect_file_id INTEGER REFERENCES files ON DELETE CASCADE;
-
-CREATE INDEX idx_files_redirect_file_id on files(redirect_file_id);
-|]
-
-down_m20240214_redirect_file_id :: Query
-down_m20240214_redirect_file_id =
-  [sql|
-DROP INDEX idx_files_redirect_file_id;
-
-ALTER TABLE files DROP COLUMN redirect_file_id;
-|]

src/Simplex/Chat/Migrations/chat_schema.sql

@@ -193,8 +193,7 @@ CREATE TABLE files(
   protocol TEXT NOT NULL DEFAULT 'smp',
   file_crypto_key BLOB,
   file_crypto_nonce BLOB,
-  note_folder_id INTEGER DEFAULT NULL REFERENCES note_folders ON DELETE CASCADE,
+  note_folder_id INTEGER DEFAULT NULL REFERENCES note_folders ON DELETE CASCADE
-  redirect_file_id INTEGER REFERENCES files ON DELETE CASCADE
 );
 CREATE TABLE snd_files(
   file_id INTEGER NOT NULL REFERENCES files ON DELETE CASCADE,
@@ -855,4 +854,3 @@ CREATE INDEX idx_chat_items_notes_item_status on chat_items(
   note_folder_id,
   item_status
 );
-CREATE INDEX idx_files_redirect_file_id on files(redirect_file_id);

src/Simplex/Chat/Store/Direct.hs

@@ -236,11 +236,18 @@ deleteContact db user@User {userId} Contact {contactId, localDisplayName, active
   if isNothing ctMember
     then do
       deleteContactProfile_ db userId contactId
-      DB.execute db "DELETE FROM display_names WHERE user_id = ? AND local_display_name = ?" (userId, localDisplayName)
+      DB.execute
+        db
+        [sql|
+          DELETE FROM display_names
+          WHERE user_id = ? AND local_display_name = ?
+            AND local_display_name NOT IN (SELECT local_display_name FROM users)
+        |]
+        (userId, localDisplayName)
     else do
       currentTs <- getCurrentTime
       DB.execute db "UPDATE group_members SET contact_id = NULL, updated_at = ? WHERE user_id = ? AND contact_id = ?" (currentTs, userId, contactId)
-  DB.execute db "DELETE FROM contacts WHERE user_id = ? AND contact_id = ?" (userId, contactId)
+  DB.execute db "DELETE FROM contacts WHERE user_id = ? AND contact_id = ? AND is_user = 0" (userId, contactId)
   forM_ activeConn $ \Connection {customUserProfileId} ->
     forM_ customUserProfileId $ \profileId ->
       deleteUnusedIncognitoProfileById_ db user profileId
@@ -250,8 +257,15 @@ deleteContactWithoutGroups :: DB.Connection -> User -> Contact -> IO ()
 deleteContactWithoutGroups db user@User {userId} Contact {contactId, localDisplayName, activeConn} = do
   DB.execute db "DELETE FROM chat_items WHERE user_id = ? AND contact_id = ?" (userId, contactId)
   deleteContactProfile_ db userId contactId
-  DB.execute db "DELETE FROM display_names WHERE user_id = ? AND local_display_name = ?" (userId, localDisplayName)
+  DB.execute
-  DB.execute db "DELETE FROM contacts WHERE user_id = ? AND contact_id = ?" (userId, contactId)
+    db
+    [sql|
+      DELETE FROM display_names
+      WHERE user_id = ? AND local_display_name = ?
+        AND local_display_name NOT IN (SELECT local_display_name FROM users)
+    |]
+    (userId, localDisplayName)
+  DB.execute db "DELETE FROM contacts WHERE user_id = ? AND contact_id = ? AND is_user = 0" (userId, contactId)
   forM_ activeConn $ \Connection {customUserProfileId} ->
     forM_ customUserProfileId $ \profileId ->
       deleteUnusedIncognitoProfileById_ db user profileId
@@ -259,7 +273,7 @@ deleteContactWithoutGroups db user@User {userId} Contact {contactId, localDispla
 setContactDeleted :: DB.Connection -> User -> Contact -> IO ()
 setContactDeleted db User {userId} Contact {contactId} = do
   currentTs <- getCurrentTime
-  DB.execute db "UPDATE contacts SET deleted = 1, updated_at = ? WHERE user_id = ? AND contact_id = ?" (currentTs, userId, contactId)
+  DB.execute db "UPDATE contacts SET deleted = 1, updated_at = ? WHERE user_id = ? AND contact_id = ? AND is_user = 0" (currentTs, userId, contactId)
 
 getDeletedContacts :: DB.Connection -> User -> IO [Contact]
 getDeletedContacts db user@User {userId} = do
@@ -501,7 +515,14 @@ updateContactLDN_ db userId contactId displayName newName updatedAt = do
     db
     "UPDATE group_members SET local_display_name = ?, updated_at = ? WHERE user_id = ? AND contact_id = ?"
     (newName, updatedAt, userId, contactId)
-  DB.execute db "DELETE FROM display_names WHERE local_display_name = ? AND user_id = ?" (displayName, userId)
+  DB.execute
+    db
+    [sql|
+      DELETE FROM display_names
+      WHERE local_display_name = ? AND user_id = ?
+        AND local_display_name NOT IN (SELECT local_display_name FROM users)
+    |]
+    (displayName, userId)
 
 getContactByName :: DB.Connection -> User -> ContactName -> ExceptT StoreError IO Contact
 getContactByName db user localDisplayName = do
@@ -614,7 +635,14 @@ createOrUpdateContactRequest db user@User {userId} userContactLinkId invId (Vers
                 WHERE user_id = ? AND contact_request_id = ?
               |]
               (invId, minV, maxV, ldn, currentTs, userId, cReqId)
-            DB.execute db "DELETE FROM display_names WHERE local_display_name = ? AND user_id = ?" (oldLdn, userId)
+            DB.execute
+              db
+              [sql|
+                DELETE FROM display_names
+                WHERE local_display_name = ? AND user_id = ?
+                  AND local_display_name NOT IN (SELECT local_display_name FROM users)
+              |]
+              (oldLdn, userId)
       where
         updateProfile currentTs =
           DB.execute
@@ -684,6 +712,7 @@ deleteContactRequest db User {userId} contactRequestId = do
         SELECT local_display_name FROM contact_requests
         WHERE user_id = ? AND contact_request_id = ?
       )
+      AND local_display_name NOT IN (SELECT local_display_name FROM users)
     |]
     (userId, userId, contactRequestId)
   DB.execute db "DELETE FROM contact_requests WHERE user_id = ? AND contact_request_id = ?" (userId, contactRequestId)

src/Simplex/Chat/Store/Files.hs

@@ -39,7 +39,6 @@ module Simplex.Chat.Store.Files
     getGroupFileIdBySharedMsgId,
     getDirectFileIdBySharedMsgId,
     getChatRefByFileId,
-    lookupChatRefByFileId,
     updateSndFileStatus,
     createSndFileChunk,
     updateSndFileChunkMsg,
@@ -47,7 +46,6 @@ module Simplex.Chat.Store.Files
     deleteSndFileChunks,
     createRcvFileTransfer,
     createRcvGroupFileTransfer,
-    createRcvStandaloneFileTransfer,
     appendRcvFD,
     getRcvFileDescrByRcvFileId,
     getRcvFileDescrBySndFileId,
@@ -72,7 +70,6 @@ module Simplex.Chat.Store.Files
     getFileTransfer,
     getFileTransferProgress,
     getFileTransferMeta,
-    lookupFileTransferRedirectMeta,
     getSndFileTransfer,
     getSndFileTransfers,
     getContactFileInfo,
@@ -89,14 +86,12 @@ import Control.Monad
 import Control.Monad.Except
 import Control.Monad.IO.Class
 import Data.Either (rights)
-import Data.Functor ((<&>))
 import Data.Int (Int64)
 import Data.Maybe (fromMaybe, isJust, listToMaybe)
 import Data.Text (Text)
 import Data.Time (addUTCTime)
 import Data.Time.Clock (UTCTime (..), getCurrentTime, nominalDay)
 import Data.Type.Equality
-import Data.Word (Word32)
 import Database.SQLite.Simple (Only (..), (:.) (..))
 import Database.SQLite.Simple.QQ (sql)
 import Database.SQLite.Simple.ToField (ToField)
@@ -189,7 +184,7 @@ createSndDirectFileTransfer db userId Contact {contactId} filePath FileInvitatio
       db
       "INSERT INTO snd_files (file_id, file_status, file_inline, connection_id, created_at, updated_at) VALUES (?,?,?,?,?,?)"
       (fileId, fileStatus, fileInline, connId, currentTs, currentTs)
-  pure FileTransferMeta {fileId, xftpSndFile = Nothing, xftpRedirectFor = Nothing, fileName, filePath, fileSize, fileInline, chunkSize, cancelled = False}
+  pure FileTransferMeta {fileId, xftpSndFile = Nothing, fileName, filePath, fileSize, fileInline, chunkSize, cancelled = False}
 
 createSndDirectFTConnection :: DB.Connection -> User -> Int64 -> (CommandId, ConnId) -> SubscriptionMode -> IO ()
 createSndDirectFTConnection db user@User {userId} fileId (cmdId, acId) subMode = do
@@ -209,7 +204,7 @@ createSndGroupFileTransfer db userId GroupInfo {groupId} filePath FileInvitation
     "INSERT INTO files (user_id, group_id, file_name, file_path, file_size, chunk_size, file_inline, ci_file_status, protocol, created_at, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)"
     ((userId, groupId, fileName, filePath, fileSize, chunkSize) :. (fileInline, CIFSSndStored, FPSMP, currentTs, currentTs))
   fileId <- insertedRowId db
-  pure FileTransferMeta {fileId, xftpSndFile = Nothing, xftpRedirectFor = Nothing, fileName, filePath, fileSize, fileInline, chunkSize, cancelled = False}
+  pure FileTransferMeta {fileId, xftpSndFile = Nothing, fileName, filePath, fileSize, fileInline, chunkSize, cancelled = False}
 
 createSndGroupFileTransferConnection :: DB.Connection -> User -> Int64 -> (CommandId, ConnId) -> GroupMember -> SubscriptionMode -> IO ()
 createSndGroupFileTransferConnection db user@User {userId} fileId (cmdId, acId) GroupMember {groupMemberId} subMode = do
@@ -282,16 +277,16 @@ getSndFTViaMsgDelivery db User {userId} Connection {connId, agentConnId} agentMs
       (\n -> SndFileTransfer {fileId, fileStatus, fileName, fileSize, chunkSize, filePath, fileDescrId, fileInline, groupMemberId, recipientDisplayName = n, connId, agentConnId})
         <$> (contactName_ <|> memberName_)
 
-createSndFileTransferXFTP :: DB.Connection -> User -> Maybe ContactOrGroup -> CryptoFile -> FileInvitation -> AgentSndFileId -> Maybe FileTransferId -> Integer -> IO FileTransferMeta
+createSndFileTransferXFTP :: DB.Connection -> User -> ContactOrGroup -> CryptoFile -> FileInvitation -> AgentSndFileId -> Integer -> IO FileTransferMeta
-createSndFileTransferXFTP db User {userId} contactOrGroup_ (CryptoFile filePath cryptoArgs) FileInvitation {fileName, fileSize} agentSndFileId xftpRedirectFor chunkSize = do
+createSndFileTransferXFTP db User {userId} contactOrGroup (CryptoFile filePath cryptoArgs) FileInvitation {fileName, fileSize} agentSndFileId chunkSize = do
   currentTs <- getCurrentTime
   let xftpSndFile = Just XFTPSndFile {agentSndFileId, privateSndFileDescr = Nothing, agentSndFileDeleted = False, cryptoArgs}
   DB.execute
     db
-    "INSERT INTO files (contact_id, group_id, user_id, file_name, file_path, file_crypto_key, file_crypto_nonce, file_size, chunk_size, redirect_file_id, agent_snd_file_id, ci_file_status, protocol, created_at, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
+    "INSERT INTO files (contact_id, group_id, user_id, file_name, file_path, file_crypto_key, file_crypto_nonce, file_size, chunk_size, agent_snd_file_id, ci_file_status, protocol, created_at, updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
-    (maybe (Nothing, Nothing) contactAndGroupIds contactOrGroup_ :. (userId, fileName, filePath, CF.fileKey <$> cryptoArgs, CF.fileNonce <$> cryptoArgs, fileSize, chunkSize) :. (xftpRedirectFor, agentSndFileId, CIFSSndStored, FPXFTP, currentTs, currentTs))
+    (contactAndGroupIds contactOrGroup :. (userId, fileName, filePath, CF.fileKey <$> cryptoArgs, CF.fileNonce <$> cryptoArgs, fileSize, chunkSize, agentSndFileId, CIFSSndStored, FPXFTP, currentTs, currentTs))
   fileId <- insertedRowId db
-  pure FileTransferMeta {fileId, xftpSndFile, xftpRedirectFor, fileName, filePath, fileSize, fileInline = Nothing, chunkSize, cancelled = False}
+  pure FileTransferMeta {fileId, xftpSndFile, fileName, filePath, fileSize, fileInline = Nothing, chunkSize, cancelled = False}
 
 createSndFTDescrXFTP :: DB.Connection -> User -> Maybe GroupMember -> Connection -> FileTransferMeta -> FileDescr -> IO ()
 createSndFTDescrXFTP db User {userId} m Connection {connId} FileTransferMeta {fileId} FileDescr {fileDescrText, fileDescrPartNo, fileDescrComplete} = do
@@ -426,14 +421,11 @@ getDirectFileIdBySharedMsgId db User {userId} Contact {contactId} sharedMsgId =
       (userId, contactId, sharedMsgId)
 
 getChatRefByFileId :: DB.Connection -> User -> Int64 -> ExceptT StoreError IO ChatRef
-getChatRefByFileId db user fileId = liftIO (lookupChatRefByFileId db user fileId) >>= maybe (throwError $ SEInternalError "could not retrieve chat ref by file id") pure
+getChatRefByFileId db User {userId} fileId =
-
+  liftIO getChatRef >>= \case
-lookupChatRefByFileId :: DB.Connection -> User -> Int64 -> IO (Maybe ChatRef)
+    [(Just contactId, Nothing)] -> pure $ ChatRef CTDirect contactId
-lookupChatRefByFileId db User {userId} fileId =
+    [(Nothing, Just groupId)] -> pure $ ChatRef CTGroup groupId
-  getChatRef <&> \case
+    _ -> throwError $ SEInternalError "could not retrieve chat ref by file id"
-    [(Just contactId, Nothing)] -> Just $ ChatRef CTDirect contactId
-    [(Nothing, Just groupId)] -> Just $ ChatRef CTGroup groupId
-    _ -> Nothing
   where
     getChatRef =
       DB.query
@@ -544,23 +536,6 @@ createRcvGroupFileTransfer db userId GroupMember {groupId, groupMemberId, localD
       (fileId, FSNew, fileConnReq, fileInline, rcvFileInline, groupMemberId, rfdId, currentTs, currentTs)
   pure RcvFileTransfer {fileId, xftpRcvFile, fileInvitation = f, fileStatus = RFSNew, rcvFileInline, senderDisplayName = c, chunkSize, cancelled = False, grpMemberId = Just groupMemberId, cryptoArgs = Nothing}
 
-createRcvStandaloneFileTransfer :: DB.Connection -> UserId -> CryptoFile -> Int64 -> Word32 -> ExceptT StoreError IO Int64
-createRcvStandaloneFileTransfer db userId (CryptoFile filePath cfArgs_) fileSize chunkSize = do
-  currentTs <- liftIO getCurrentTime
-  fileId <- liftIO $ do
-    DB.execute
-      db
-      "INSERT INTO files (user_id, file_name, file_path, file_size, chunk_size, ci_file_status, protocol, created_at, updated_at) VALUES (?,?,?,?,?,?,?,?,?)"
-      (userId, takeFileName filePath, filePath, fileSize, chunkSize, CIFSRcvInvitation, FPXFTP, currentTs, currentTs)
-    insertedRowId db
-  liftIO . forM_ cfArgs_ $ \cfArgs -> setFileCryptoArgs_ db fileId cfArgs currentTs
-  liftIO $
-    DB.execute
-      db
-      "INSERT INTO rcv_files (file_id, file_status, created_at, updated_at) VALUES (?,?,?,?)"
-      (fileId, FSNew, currentTs, currentTs)
-  pure fileId
-
 createRcvFD_ :: DB.Connection -> UserId -> UTCTime -> FileDescr -> ExceptT StoreError IO RcvFileDescr
 createRcvFD_ db userId currentTs FileDescr {fileDescrText, fileDescrPartNo, fileDescrComplete} = do
   when (fileDescrPartNo /= 0) $ throwError SERcvFileInvalidDescrPart
@@ -687,9 +662,9 @@ getRcvFileTransfer_ db userId fileId = do
       (FileStatus, Maybe ConnReqInvitation, Maybe Int64, String, Integer, Integer, Maybe Bool) :. (Maybe ContactName, Maybe ContactName, Maybe FilePath, Maybe C.SbKey, Maybe C.CbNonce, Maybe InlineFileMode, Maybe InlineFileMode, Maybe AgentRcvFileId, Bool) :. (Maybe Int64, Maybe AgentConnId) ->
       ExceptT StoreError IO RcvFileTransfer
     rcvFileTransfer rfd_ ((fileStatus', fileConnReq, grpMemberId, fileName, fileSize, chunkSize, cancelled_) :. (contactName_, memberName_, filePath_, fileKey, fileNonce, fileInline, rcvFileInline, agentRcvFileId, agentRcvFileDeleted) :. (connId_, agentConnId_)) =
-      case contactName_ <|> memberName_ <|> standaloneName_ of
+      case contactName_ <|> memberName_ of
         Nothing -> throwError $ SERcvFileInvalid fileId
-        Just name ->
+        Just name -> do
           case fileStatus' of
             FSNew -> pure $ ft name RFSNew
             FSAccepted -> ft name . RFSAccepted <$> rfi
@@ -697,9 +672,6 @@ getRcvFileTransfer_ db userId fileId = do
             FSComplete -> ft name . RFSComplete <$> rfi
             FSCancelled -> ft name . RFSCancelled <$> rfi_
       where
-        standaloneName_ = case (connId_, agentRcvFileId, filePath_) of
-          (Nothing, Just _, Just _) -> Just "" -- filePath marks files that are accepted from contact or, in this case, set by createRcvDirectFileTransfer
-          _ -> Nothing
         ft senderDisplayName fileStatus =
           let fileInvitation = FileInvitation {fileName, fileSize, fileDigest = Nothing, fileConnReq, fileInline, fileDescr = Nothing}
               cryptoArgs = CFArgs <$> fileKey <*> fileNonce
@@ -934,22 +906,17 @@ getFileTransferMeta_ db userId fileId =
     DB.query
       db
       [sql|
-        SELECT file_name, file_size, chunk_size, file_path, file_crypto_key, file_crypto_nonce, file_inline, agent_snd_file_id, agent_snd_file_deleted, private_snd_file_descr, cancelled, redirect_file_id
+        SELECT file_name, file_size, chunk_size, file_path, file_crypto_key, file_crypto_nonce, file_inline, agent_snd_file_id, agent_snd_file_deleted, private_snd_file_descr, cancelled
         FROM files
         WHERE user_id = ? AND file_id = ?
       |]
       (userId, fileId)
   where
-    fileTransferMeta :: (String, Integer, Integer, FilePath, Maybe C.SbKey, Maybe C.CbNonce, Maybe InlineFileMode, Maybe AgentSndFileId, Bool, Maybe Text, Maybe Bool, Maybe FileTransferId) -> FileTransferMeta
+    fileTransferMeta :: (String, Integer, Integer, FilePath, Maybe C.SbKey, Maybe C.CbNonce, Maybe InlineFileMode, Maybe AgentSndFileId, Bool, Maybe Text, Maybe Bool) -> FileTransferMeta
-    fileTransferMeta (fileName, fileSize, chunkSize, filePath, fileKey, fileNonce, fileInline, aSndFileId_, agentSndFileDeleted, privateSndFileDescr, cancelled_, xftpRedirectFor) =
+    fileTransferMeta (fileName, fileSize, chunkSize, filePath, fileKey, fileNonce, fileInline, aSndFileId_, agentSndFileDeleted, privateSndFileDescr, cancelled_) =
       let cryptoArgs = CFArgs <$> fileKey <*> fileNonce
           xftpSndFile = (\fId -> XFTPSndFile {agentSndFileId = fId, privateSndFileDescr, agentSndFileDeleted, cryptoArgs}) <$> aSndFileId_
-        in FileTransferMeta {fileId, xftpSndFile, xftpRedirectFor, fileName, fileSize, chunkSize, filePath, fileInline, cancelled = fromMaybe False cancelled_}
+       in FileTransferMeta {fileId, xftpSndFile, fileName, fileSize, chunkSize, filePath, fileInline, cancelled = fromMaybe False cancelled_}
-
-lookupFileTransferRedirectMeta :: DB.Connection -> User -> Int64 -> IO [FileTransferMeta]
-lookupFileTransferRedirectMeta db User {userId} fileId = do
-  redirects <- DB.query db "SELECT file_id FROM files WHERE user_id = ? AND redirect_file_id = ?" (userId, fileId)
-  rights <$> mapM (runExceptT . getFileTransferMeta_ db userId . fromOnly) redirects
 
 createLocalFile :: ToField (CIFileStatus d) => CIFileStatus d -> DB.Connection -> User -> NoteFolder -> ChatItemId -> UTCTime -> CryptoFile -> Integer -> Integer -> IO Int64
 createLocalFile fileStatus db User {userId} NoteFolder {noteFolderId} chatItemId itemTs CryptoFile {filePath, cryptoArgs} fileSize fileChunkSize = do

src/Simplex/Chat/Store/Groups.hs

@@ -225,6 +225,7 @@ deleteGroupLink db User {userId} GroupInfo {groupId} = do
           JOIN user_contact_links uc USING (user_contact_link_id)
           WHERE uc.user_id = ? AND uc.group_id = ?
         )
+        AND local_display_name NOT IN (SELECT local_display_name FROM users)
     |]
     (userId, userId, groupId)
   DB.execute
@@ -586,7 +587,14 @@ deleteGroup :: DB.Connection -> User -> GroupInfo -> IO ()
 deleteGroup db user@User {userId} g@GroupInfo {groupId, localDisplayName} = do
   deleteGroupProfile_ db userId groupId
   DB.execute db "DELETE FROM groups WHERE user_id = ? AND group_id = ?" (userId, groupId)
-  DB.execute db "DELETE FROM display_names WHERE user_id = ? AND local_display_name = ?" (userId, localDisplayName)
+  DB.execute
+    db
+    [sql|
+      DELETE FROM display_names
+      WHERE user_id = ? AND local_display_name = ?
+        AND local_display_name NOT IN (SELECT local_display_name FROM users)
+    |]
+    (userId, localDisplayName)
   forM_ (incognitoMembershipProfile g) $ deleteUnusedIncognitoProfileById_ db user . localProfileId
 
 deleteGroupProfile_ :: DB.Connection -> UserId -> GroupId -> IO ()
@@ -1051,7 +1059,14 @@ cleanupMemberProfileAndName_ db User {userId} GroupMember {groupMemberId, member
     sameProfileMember :: (Maybe GroupMemberId) <- maybeFirstRow fromOnly $ DB.query db "SELECT group_member_id FROM group_members WHERE user_id = ? AND contact_profile_id = ? AND group_member_id != ? LIMIT 1" (userId, memberContactProfileId, groupMemberId)
     when (isNothing sameProfileMember) $ do
       DB.execute db "DELETE FROM contact_profiles WHERE user_id = ? AND contact_profile_id = ?" (userId, memberContactProfileId)
-      DB.execute db "DELETE FROM display_names WHERE user_id = ? AND local_display_name = ?" (userId, localDisplayName)
+      DB.execute
+        db
+        [sql|
+          DELETE FROM display_names
+          WHERE user_id = ? AND local_display_name = ?
+            AND local_display_name NOT IN (SELECT local_display_name FROM users)
+        |]
+        (userId, localDisplayName)
 
 deleteGroupMemberConnection :: DB.Connection -> User -> GroupMember -> IO ()
 deleteGroupMemberConnection db User {userId} GroupMember {groupMemberId} =
@@ -1361,7 +1376,14 @@ updateGroupProfile db User {userId} g@GroupInfo {groupId, localDisplayName, grou
         db
         "UPDATE groups SET local_display_name = ?, updated_at = ? WHERE user_id = ? AND group_id = ?"
         (ldn, currentTs, userId, groupId)
-      DB.execute db "DELETE FROM display_names WHERE local_display_name = ? AND user_id = ?" (localDisplayName, userId)
+      DB.execute
+        db
+        [sql|
+          DELETE FROM display_names
+          WHERE local_display_name = ? AND user_id = ?
+            AND local_display_name NOT IN (SELECT local_display_name FROM users)
+        |]
+        (localDisplayName, userId)
 
 getGroupInfo :: DB.Connection -> VersionRange -> User -> Int64 -> ExceptT StoreError IO GroupInfo
 getGroupInfo db vr User {userId, userContactId} groupId =
@@ -1464,7 +1486,7 @@ getMatchingContacts db user@User {userId} Contact {contactId, profile = LocalPro
         FROM contacts ct
         JOIN contact_profiles p ON ct.contact_profile_id = p.contact_profile_id
         WHERE ct.user_id = ? AND ct.contact_id != ?
-          AND ct.contact_status = ? AND ct.deleted = 0
+          AND ct.contact_status = ? AND ct.deleted = 0 AND ct.is_user = 0
           AND p.display_name = ? AND p.full_name = ?
       |]
 
@@ -1502,7 +1524,7 @@ getMatchingMemberContacts db user@User {userId} GroupMember {memberProfile = Loc
         FROM contacts ct
         JOIN contact_profiles p ON ct.contact_profile_id = p.contact_profile_id
         WHERE ct.user_id = ?
-          AND ct.contact_status = ? AND ct.deleted = 0
+          AND ct.contact_status = ? AND ct.deleted = 0 AND ct.is_user = 0
           AND p.display_name = ? AND p.full_name = ?
       |]
 
@@ -1656,7 +1678,7 @@ mergeContactRecords db user@User {userId} to@Contact {localDisplayName = keepLDN
         ":updated_at" := currentTs
       ]
     deleteContactProfile_ db userId fromContactId
-    DB.execute db "DELETE FROM contacts WHERE contact_id = ? AND user_id = ?" (fromContactId, userId)
+    DB.execute db "DELETE FROM contacts WHERE contact_id = ? AND user_id = ? AND is_user = 0" (fromContactId, userId)
     deleteUnusedDisplayName_ db userId fromLDN
     when (keepLDN /= toLDN && keepLDN == fromLDN) $
       DB.execute
@@ -2030,7 +2052,14 @@ updateMemberProfile db User {userId} m p'
           db
           "UPDATE group_members SET local_display_name = ?, updated_at = ? WHERE user_id = ? AND group_member_id = ?"
           (ldn, currentTs, userId, groupMemberId)
-        DB.execute db "DELETE FROM display_names WHERE local_display_name = ? AND user_id = ?" (localDisplayName, userId)
+        DB.execute
+          db
+          [sql|
+            DELETE FROM display_names
+            WHERE local_display_name = ? AND user_id = ?
+              AND local_display_name NOT IN (SELECT local_display_name FROM users)
+          |]
+          (localDisplayName, userId)
         pure $ Right m {localDisplayName = ldn, memberProfile = profile}
   where
     GroupMember {groupMemberId, localDisplayName, memberProfile = LocalProfile {profileId, displayName, localAlias}} = m

src/Simplex/Chat/Store/Messages.hs

@@ -39,7 +39,7 @@ module Simplex.Chat.Store.Messages
     getDirectChat,
     getGroupChat,
     getLocalChat,
-    getDirectChatItemLast,
+    getDirectChatItemsLast,
     getAllChatItems,
     getAChatItem,
     updateDirectChatItem,
@@ -92,7 +92,6 @@ module Simplex.Chat.Store.Messages
     getLocalChatItemIdByText,
     getLocalChatItemIdByText',
     getChatItemByFileId,
-    lookupChatItemByFileId,
     getChatItemByGroupId,
     updateDirectChatItemStatus,
     getTimedItems,
@@ -126,7 +125,6 @@ import Data.List (sortBy)
 import Data.Maybe (fromMaybe, isJust, mapMaybe)
 import Data.Ord (Down (..), comparing)
 import Data.Text (Text)
-import qualified Data.Text as T
 import Data.Time (addUTCTime)
 import Data.Time.Clock (UTCTime (..), getCurrentTime)
 import Database.SQLite.Simple (NamedParam (..), Only (..), Query, (:.) (..))
@@ -830,7 +828,7 @@ toLocalChatItem currentTs ((itemId, itemTs, AMsgDirection msgDir, itemContentTex
     cItem :: MsgDirectionI d => SMsgDirection d -> CIDirection 'CTLocal d -> CIStatus d -> CIContent d -> Maybe (CIFile d) -> CChatItem 'CTLocal
     cItem d chatDir ciStatus content file =
       CChatItem d ChatItem {chatDir, meta = ciMeta content ciStatus, content, formattedText = parseMaybeMarkdownList itemText, quotedItem = Nothing, reactions = [], file}
-    badItem = Left $ SEBadChatItem itemId (Just itemTs)
+    badItem = Left $ SEBadChatItem itemId
     ciMeta :: CIContent d -> CIStatus d -> CIMeta 'CTLocal d
     ciMeta content status =
       let itemDeleted' = case itemDeleted of
@@ -924,118 +922,97 @@ getDirectChat :: DB.Connection -> User -> Int64 -> ChatPagination -> Maybe Strin
 getDirectChat db user contactId pagination search_ = do
   let search = fromMaybe "" search_
   ct <- getContact db user contactId
-  liftIO $ case pagination of
+  liftIO . getDirectChatReactions_ db ct =<< case pagination of
     CPLast count -> getDirectChatLast_ db user ct count search
     CPAfter afterId count -> getDirectChatAfter_ db user ct afterId count search
     CPBefore beforeId count -> getDirectChatBefore_ db user ct beforeId count search
 
--- the last items in reverse order (the last item in the conversation is the first in the returned list)
+getDirectChatLast_ :: DB.Connection -> User -> Contact -> Int -> String -> ExceptT StoreError IO (Chat 'CTDirect)
-getDirectChatLast_ :: DB.Connection -> User -> Contact -> Int -> String -> IO (Chat 'CTDirect)
+getDirectChatLast_ db user ct@Contact {contactId} count search = do
-getDirectChatLast_ db user@User {userId} ct@Contact {contactId} count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getDirectChatItemIdsLast_
+  chatItems <- getDirectChatItemsLast db user contactId count search
-  currentTs <- getCurrentTime
-  chatItems <- mapM (safeGetDirectItem db user ct currentTs) chatItemIds
   pure $ Chat (DirectChat ct) (reverse chatItems) stats
-  where
-    getDirectChatItemIdsLast_ :: IO [ChatItemId]
-    getDirectChatItemIdsLast_ =
-      map fromOnly
-        <$> DB.query
-          db
-          [sql|
-            SELECT chat_item_id
-            FROM chat_items
-            WHERE user_id = ? AND contact_id = ? AND item_text LIKE '%' || ? || '%'
-            ORDER BY created_at DESC, chat_item_id DESC
-            LIMIT ?
-          |]
-          (userId, contactId, search, count)
 
-safeGetDirectItem :: DB.Connection -> User -> Contact -> UTCTime -> ChatItemId -> IO (CChatItem 'CTDirect)
+-- the last items in reverse order (the last item in the conversation is the first in the returned list)
-safeGetDirectItem db user ct currentTs itemId =
+getDirectChatItemsLast :: DB.Connection -> User -> ContactId -> Int -> String -> ExceptT StoreError IO [CChatItem 'CTDirect]
-  runExceptT (getDirectCIWithReactions db user ct itemId)
+getDirectChatItemsLast db User {userId} contactId count search = ExceptT $ do
-    >>= pure <$> safeToDirectItem currentTs itemId
-
-safeToDirectItem :: UTCTime -> ChatItemId -> Either StoreError (CChatItem 'CTDirect) -> CChatItem 'CTDirect
-safeToDirectItem currentTs itemId = \case
-  Right ci -> ci
-  Left e@(SEBadChatItem _ (Just itemTs)) -> badDirectItem itemTs e
-  Left e -> badDirectItem currentTs e
-  where
-    badDirectItem :: UTCTime -> StoreError -> CChatItem 'CTDirect
-    badDirectItem ts e =
-      let errorText = T.pack $ show e
-       in CChatItem
-            SMDSnd
-            ChatItem
-              { chatDir = CIDirectSnd,
-                meta = dummyMeta itemId ts errorText,
-                content = CIInvalidJSON errorText,
-                formattedText = Nothing,
-                quotedItem = Nothing,
-                reactions = [],
-                file = Nothing
-              }
-
-getDirectChatItemLast :: DB.Connection -> User -> ContactId -> ExceptT StoreError IO (CChatItem 'CTDirect)
-getDirectChatItemLast db user@User {userId} contactId = do
-  chatItemId <-
-    ExceptT . firstRow fromOnly (SEChatItemNotFoundByContactId contactId) $
-      DB.query
-        db
-        [sql|
-          SELECT chat_item_id
-          FROM chat_items
-          WHERE user_id = ? AND contact_id = ?
-          ORDER BY created_at DESC, chat_item_id DESC
-          LIMIT 1
-        |]
-        (userId, contactId)
-  getDirectChatItem db user contactId chatItemId
-
-getDirectChatAfter_ :: DB.Connection -> User -> Contact -> ChatItemId -> Int -> String -> IO (Chat 'CTDirect)
-getDirectChatAfter_ db user@User {userId} ct@Contact {contactId} afterChatItemId count search = do
-  let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getDirectChatItemIdsAfter_
   currentTs <- getCurrentTime
-  chatItems <- mapM (safeGetDirectItem db user ct currentTs) chatItemIds
+  mapM (toDirectChatItem currentTs)
+    <$> DB.query
+      db
+      [sql|
+        SELECT
+          -- ChatItem
+          i.chat_item_id, i.item_ts, i.item_sent, i.item_content, i.item_text, i.item_status, i.shared_msg_id, i.item_deleted, i.item_deleted_ts, i.item_edited, i.created_at, i.updated_at, i.timed_ttl, i.timed_delete_at, i.item_live,
+          -- CIFile
+          f.file_id, f.file_name, f.file_size, f.file_path, f.file_crypto_key, f.file_crypto_nonce, f.ci_file_status, f.protocol,
+          -- DirectQuote
+          ri.chat_item_id, i.quoted_shared_msg_id, i.quoted_sent_at, i.quoted_content, i.quoted_sent
+        FROM chat_items i
+        LEFT JOIN files f ON f.chat_item_id = i.chat_item_id
+        LEFT JOIN chat_items ri ON ri.user_id = i.user_id AND ri.contact_id = i.contact_id AND ri.shared_msg_id = i.quoted_shared_msg_id
+        WHERE i.user_id = ? AND i.contact_id = ? AND i.item_text LIKE '%' || ? || '%'
+        ORDER BY i.created_at DESC, i.chat_item_id DESC
+        LIMIT ?
+      |]
+      (userId, contactId, search, count)
+
+getDirectChatAfter_ :: DB.Connection -> User -> Contact -> ChatItemId -> Int -> String -> ExceptT StoreError IO (Chat 'CTDirect)
+getDirectChatAfter_ db User {userId} ct@Contact {contactId} afterChatItemId count search = do
+  let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
+  chatItems <- ExceptT getDirectChatItemsAfter_
   pure $ Chat (DirectChat ct) chatItems stats
   where
-    getDirectChatItemIdsAfter_ :: IO [ChatItemId]
+    getDirectChatItemsAfter_ :: IO (Either StoreError [CChatItem 'CTDirect])
-    getDirectChatItemIdsAfter_ =
+    getDirectChatItemsAfter_ = do
-      map fromOnly
+      currentTs <- getCurrentTime
+      mapM (toDirectChatItem currentTs)
         <$> DB.query
           db
           [sql|
-            SELECT chat_item_id
+            SELECT
-            FROM chat_items
+              -- ChatItem
-            WHERE user_id = ? AND contact_id = ? AND item_text LIKE '%' || ? || '%'
+              i.chat_item_id, i.item_ts, i.item_sent, i.item_content, i.item_text, i.item_status, i.shared_msg_id, i.item_deleted, i.item_deleted_ts, i.item_edited, i.created_at, i.updated_at, i.timed_ttl, i.timed_delete_at, i.item_live,
-              AND chat_item_id > ?
+              -- CIFile
-            ORDER BY created_at ASC, chat_item_id ASC
+              f.file_id, f.file_name, f.file_size, f.file_path, f.file_crypto_key, f.file_crypto_nonce, f.ci_file_status, f.protocol,
+              -- DirectQuote
+              ri.chat_item_id, i.quoted_shared_msg_id, i.quoted_sent_at, i.quoted_content, i.quoted_sent
+            FROM chat_items i
+            LEFT JOIN files f ON f.chat_item_id = i.chat_item_id
+            LEFT JOIN chat_items ri ON ri.user_id = i.user_id AND ri.contact_id = i.contact_id AND ri.shared_msg_id = i.quoted_shared_msg_id
+            WHERE i.user_id = ? AND i.contact_id = ? AND i.item_text LIKE '%' || ? || '%'
+              AND i.chat_item_id > ?
+            ORDER BY i.created_at ASC, i.chat_item_id ASC
             LIMIT ?
           |]
           (userId, contactId, search, afterChatItemId, count)
 
-getDirectChatBefore_ :: DB.Connection -> User -> Contact -> ChatItemId -> Int -> String -> IO (Chat 'CTDirect)
+getDirectChatBefore_ :: DB.Connection -> User -> Contact -> ChatItemId -> Int -> String -> ExceptT StoreError IO (Chat 'CTDirect)
-getDirectChatBefore_ db user@User {userId} ct@Contact {contactId} beforeChatItemId count search = do
+getDirectChatBefore_ db User {userId} ct@Contact {contactId} beforeChatItemId count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getDirectChatItemsIdsBefore_
+  chatItems <- ExceptT getDirectChatItemsBefore_
-  currentTs <- getCurrentTime
-  chatItems <- mapM (safeGetDirectItem db user ct currentTs) chatItemIds
   pure $ Chat (DirectChat ct) (reverse chatItems) stats
   where
-    getDirectChatItemsIdsBefore_ :: IO [ChatItemId]
+    getDirectChatItemsBefore_ :: IO (Either StoreError [CChatItem 'CTDirect])
-    getDirectChatItemsIdsBefore_ =
+    getDirectChatItemsBefore_ = do
-      map fromOnly
+      currentTs <- getCurrentTime
+      mapM (toDirectChatItem currentTs)
         <$> DB.query
           db
           [sql|
-            SELECT chat_item_id
+            SELECT
-            FROM chat_items
+              -- ChatItem
-            WHERE user_id = ? AND contact_id = ? AND item_text LIKE '%' || ? || '%'
+              i.chat_item_id, i.item_ts, i.item_sent, i.item_content, i.item_text, i.item_status, i.shared_msg_id, i.item_deleted, i.item_deleted_ts, i.item_edited, i.created_at, i.updated_at, i.timed_ttl, i.timed_delete_at, i.item_live,
-              AND chat_item_id < ?
+              -- CIFile
-            ORDER BY created_at DESC, chat_item_id DESC
+              f.file_id, f.file_name, f.file_size, f.file_path, f.file_crypto_key, f.file_crypto_nonce, f.ci_file_status, f.protocol,
+              -- DirectQuote
+              ri.chat_item_id, i.quoted_shared_msg_id, i.quoted_sent_at, i.quoted_content, i.quoted_sent
+            FROM chat_items i
+            LEFT JOIN files f ON f.chat_item_id = i.chat_item_id
+            LEFT JOIN chat_items ri ON ri.user_id = i.user_id AND ri.contact_id = i.contact_id AND ri.shared_msg_id = i.quoted_shared_msg_id
+            WHERE i.user_id = ? AND i.contact_id = ? AND i.item_text LIKE '%' || ? || '%'
+              AND i.chat_item_id < ?
+            ORDER BY i.created_at DESC, i.chat_item_id DESC
             LIMIT ?
           |]
           (userId, contactId, search, beforeChatItemId, count)
@@ -1045,16 +1022,15 @@ getGroupChat db vr user groupId pagination search_ = do
   let search = fromMaybe "" search_
   g <- getGroupInfo db vr user groupId
   case pagination of
-    CPLast count -> liftIO $ getGroupChatLast_ db user g count search
+    CPLast count -> getGroupChatLast_ db user g count search
     CPAfter afterId count -> getGroupChatAfter_ db user g afterId count search
     CPBefore beforeId count -> getGroupChatBefore_ db user g beforeId count search
 
-getGroupChatLast_ :: DB.Connection -> User -> GroupInfo -> Int -> String -> IO (Chat 'CTGroup)
+getGroupChatLast_ :: DB.Connection -> User -> GroupInfo -> Int -> String -> ExceptT StoreError IO (Chat 'CTGroup)
 getGroupChatLast_ db user@User {userId} g@GroupInfo {groupId} count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getGroupChatItemIdsLast_
+  chatItemIds <- liftIO getGroupChatItemIdsLast_
-  currentTs <- getCurrentTime
+  chatItems <- mapM (getGroupCIWithReactions db user g) chatItemIds
-  chatItems <- mapM (safeGetGroupItem db user g currentTs) chatItemIds
   pure $ Chat (GroupChat g) (reverse chatItems) stats
   where
     getGroupChatItemIdsLast_ :: IO [ChatItemId]
@@ -1071,32 +1047,6 @@ getGroupChatLast_ db user@User {userId} g@GroupInfo {groupId} count search = do
           |]
           (userId, groupId, search, count)
 
-safeGetGroupItem :: DB.Connection -> User -> GroupInfo -> UTCTime -> ChatItemId -> IO (CChatItem 'CTGroup)
-safeGetGroupItem db user g currentTs itemId =
-  runExceptT (getGroupCIWithReactions db user g itemId)
-    >>= pure <$> safeToGroupItem currentTs itemId
-
-safeToGroupItem :: UTCTime -> ChatItemId -> Either StoreError (CChatItem 'CTGroup) -> CChatItem 'CTGroup
-safeToGroupItem currentTs itemId = \case
-  Right ci -> ci
-  Left e@(SEBadChatItem _ (Just itemTs)) -> badGroupItem itemTs e
-  Left e -> badGroupItem currentTs e
-  where
-    badGroupItem :: UTCTime -> StoreError -> CChatItem 'CTGroup
-    badGroupItem ts e =
-      let errorText = T.pack $ show e
-       in CChatItem
-            SMDSnd
-            ChatItem
-              { chatDir = CIGroupSnd,
-                meta = dummyMeta itemId ts errorText,
-                content = CIInvalidJSON errorText,
-                formattedText = Nothing,
-                quotedItem = Nothing,
-                reactions = [],
-                file = Nothing
-              }
-
 getGroupMemberChatItemLast :: DB.Connection -> User -> GroupId -> GroupMemberId -> ExceptT StoreError IO (CChatItem 'CTGroup)
 getGroupMemberChatItemLast db user@User {userId} groupId groupMemberId = do
   chatItemId <-
@@ -1118,8 +1068,7 @@ getGroupChatAfter_ db user@User {userId} g@GroupInfo {groupId} afterChatItemId c
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
   afterChatItem <- getGroupChatItem db user groupId afterChatItemId
   chatItemIds <- liftIO $ getGroupChatItemIdsAfter_ (chatItemTs afterChatItem)
-  currentTs <- liftIO getCurrentTime
+  chatItems <- mapM (getGroupCIWithReactions db user g) chatItemIds
-  chatItems <- liftIO $ mapM (safeGetGroupItem db user g currentTs) chatItemIds
   pure $ Chat (GroupChat g) chatItems stats
   where
     getGroupChatItemIdsAfter_ :: UTCTime -> IO [ChatItemId]
@@ -1142,8 +1091,7 @@ getGroupChatBefore_ db user@User {userId} g@GroupInfo {groupId} beforeChatItemId
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
   beforeChatItem <- getGroupChatItem db user groupId beforeChatItemId
   chatItemIds <- liftIO $ getGroupChatItemIdsBefore_ (chatItemTs beforeChatItem)
-  currentTs <- liftIO getCurrentTime
+  chatItems <- mapM (getGroupCIWithReactions db user g) chatItemIds
-  chatItems <- liftIO $ mapM (safeGetGroupItem db user g currentTs) chatItemIds
   pure $ Chat (GroupChat g) (reverse chatItems) stats
   where
     getGroupChatItemIdsBefore_ :: UTCTime -> IO [ChatItemId]
@@ -1165,17 +1113,16 @@ getLocalChat :: DB.Connection -> User -> Int64 -> ChatPagination -> Maybe String
 getLocalChat db user folderId pagination search_ = do
   let search = fromMaybe "" search_
   nf <- getNoteFolder db user folderId
-  liftIO $ case pagination of
+  case pagination of
     CPLast count -> getLocalChatLast_ db user nf count search
     CPAfter afterId count -> getLocalChatAfter_ db user nf afterId count search
     CPBefore beforeId count -> getLocalChatBefore_ db user nf beforeId count search
 
-getLocalChatLast_ :: DB.Connection -> User -> NoteFolder -> Int -> String -> IO (Chat 'CTLocal)
+getLocalChatLast_ :: DB.Connection -> User -> NoteFolder -> Int -> String -> ExceptT StoreError IO (Chat 'CTLocal)
 getLocalChatLast_ db user@User {userId} nf@NoteFolder {noteFolderId} count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getLocalChatItemIdsLast_
+  chatItemIds <- liftIO getLocalChatItemIdsLast_
-  currentTs <- getCurrentTime
+  chatItems <- mapM (getLocalChatItem db user noteFolderId) chatItemIds
-  chatItems <- mapM (safeGetLocalItem db user nf currentTs) chatItemIds
   pure $ Chat (LocalChat nf) (reverse chatItems) stats
   where
     getLocalChatItemIdsLast_ :: IO [ChatItemId]
@@ -1192,38 +1139,11 @@ getLocalChatLast_ db user@User {userId} nf@NoteFolder {noteFolderId} count searc
           |]
           (userId, noteFolderId, search, count)
 
-safeGetLocalItem :: DB.Connection -> User -> NoteFolder -> UTCTime -> ChatItemId -> IO (CChatItem 'CTLocal)
+getLocalChatAfter_ :: DB.Connection -> User -> NoteFolder -> ChatItemId -> Int -> String -> ExceptT StoreError IO (Chat 'CTLocal)
-safeGetLocalItem db user NoteFolder {noteFolderId} currentTs itemId =
-  runExceptT (getLocalChatItem db user noteFolderId itemId)
-    >>= pure <$> safeToLocalItem currentTs itemId
-
-safeToLocalItem :: UTCTime -> ChatItemId -> Either StoreError (CChatItem 'CTLocal) -> CChatItem 'CTLocal
-safeToLocalItem currentTs itemId = \case
-  Right ci -> ci
-  Left e@(SEBadChatItem _ (Just itemTs)) -> badLocalItem itemTs e
-  Left e -> badLocalItem currentTs e
-  where
-    badLocalItem :: UTCTime -> StoreError -> CChatItem 'CTLocal
-    badLocalItem ts e =
-      let errorText = T.pack $ show e
-       in CChatItem
-            SMDSnd
-            ChatItem
-              { chatDir = CILocalSnd,
-                meta = dummyMeta itemId ts errorText,
-                content = CIInvalidJSON errorText,
-                formattedText = Nothing,
-                quotedItem = Nothing,
-                reactions = [],
-                file = Nothing
-              }
-
-getLocalChatAfter_ :: DB.Connection -> User -> NoteFolder -> ChatItemId -> Int -> String -> IO (Chat 'CTLocal)
 getLocalChatAfter_ db user@User {userId} nf@NoteFolder {noteFolderId} afterChatItemId count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getLocalChatItemIdsAfter_
+  chatItemIds <- liftIO getLocalChatItemIdsAfter_
-  currentTs <- getCurrentTime
+  chatItems <- mapM (getLocalChatItem db user noteFolderId) chatItemIds
-  chatItems <- mapM (safeGetLocalItem db user nf currentTs) chatItemIds
   pure $ Chat (LocalChat nf) chatItems stats
   where
     getLocalChatItemIdsAfter_ :: IO [ChatItemId]
@@ -1241,12 +1161,11 @@ getLocalChatAfter_ db user@User {userId} nf@NoteFolder {noteFolderId} afterChatI
           |]
           (userId, noteFolderId, search, afterChatItemId, count)
 
-getLocalChatBefore_ :: DB.Connection -> User -> NoteFolder -> ChatItemId -> Int -> String -> IO (Chat 'CTLocal)
+getLocalChatBefore_ :: DB.Connection -> User -> NoteFolder -> ChatItemId -> Int -> String -> ExceptT StoreError IO (Chat 'CTLocal)
 getLocalChatBefore_ db user@User {userId} nf@NoteFolder {noteFolderId} beforeChatItemId count search = do
   let stats = ChatStats {unreadCount = 0, minUnreadItemId = 0, unreadChat = False}
-  chatItemIds <- getLocalChatItemIdsBefore_
+  chatItemIds <- liftIO getLocalChatItemIdsBefore_
-  currentTs <- getCurrentTime
+  chatItems <- mapM (getLocalChatItem db user noteFolderId) chatItemIds
-  chatItems <- mapM (safeGetLocalItem db user nf currentTs) chatItemIds
   pure $ Chat (LocalChat nf) (reverse chatItems) stats
   where
     getLocalChatItemIdsBefore_ :: IO [ChatItemId]
@@ -1269,7 +1188,7 @@ toChatItemRef = \case
   (itemId, Just contactId, Nothing, Nothing) -> Right (ChatRef CTDirect contactId, itemId)
   (itemId, Nothing, Just groupId, Nothing) -> Right (ChatRef CTGroup groupId, itemId)
   (itemId, Nothing, Nothing, Just folderId) -> Right (ChatRef CTLocal folderId, itemId)
-  (itemId, _, _, _) -> Left $ SEBadChatItem itemId Nothing
+  (itemId, _, _, _) -> Left $ SEBadChatItem itemId
 
 updateDirectChatItemsRead :: DB.Connection -> User -> ContactId -> Maybe (ChatItemId, ChatItemId) -> IO ()
 updateDirectChatItemsRead db User {userId} contactId itemsRange_ = do
@@ -1442,7 +1361,7 @@ toDirectChatItem currentTs (((itemId, itemTs, AMsgDirection msgDir, itemContentT
     cItem :: MsgDirectionI d => SMsgDirection d -> CIDirection 'CTDirect d -> CIStatus d -> CIContent d -> Maybe (CIFile d) -> CChatItem 'CTDirect
     cItem d chatDir ciStatus content file =
       CChatItem d ChatItem {chatDir, meta = ciMeta content ciStatus, content, formattedText = parseMaybeMarkdownList itemText, quotedItem = toDirectQuote quoteRow, reactions = [], file}
-    badItem = Left $ SEBadChatItem itemId (Just itemTs)
+    badItem = Left $ SEBadChatItem itemId
     ciMeta :: CIContent d -> CIStatus d -> CIMeta 'CTDirect d
     ciMeta content status =
       let itemDeleted' = case itemDeleted of
@@ -1493,7 +1412,7 @@ toGroupChatItem currentTs userContactId (((itemId, itemTs, AMsgDirection msgDir,
     cItem :: MsgDirectionI d => SMsgDirection d -> CIDirection 'CTGroup d -> CIStatus d -> CIContent d -> Maybe (CIFile d) -> CChatItem 'CTGroup
     cItem d chatDir ciStatus content file =
       CChatItem d ChatItem {chatDir, meta = ciMeta content ciStatus, content, formattedText = parseMaybeMarkdownList itemText, quotedItem = toGroupQuote quoteRow quotedMember_, reactions = [], file}
-    badItem = Left $ SEBadChatItem itemId (Just itemTs)
+    badItem = Left $ SEBadChatItem itemId
     ciMeta :: CIContent d -> CIStatus d -> CIMeta 'CTGroup d
     ciMeta content status =
       let itemDeleted' = case itemDeleted of
@@ -2166,12 +2085,6 @@ getChatItemByFileId db vr user@User {userId} fileId = do
         (userId, fileId)
   getAChatItem db vr user chatRef itemId
 
-lookupChatItemByFileId :: DB.Connection -> VersionRange -> User -> Int64 -> ExceptT StoreError IO (Maybe AChatItem)
-lookupChatItemByFileId db vr user fileId = do
-  fmap Just (getChatItemByFileId db vr user fileId) `catchError` \case
-    SEChatItemNotFoundByFileId {} -> pure Nothing
-    e -> throwError e
-
 getChatItemByGroupId :: DB.Connection -> VersionRange -> User -> GroupId -> ExceptT StoreError IO AChatItem
 getChatItemByGroupId db vr user@User {userId} groupId = do
   (chatRef, itemId) <-
@@ -2196,7 +2109,7 @@ getChatRefViaItemId db User {userId} itemId = do
     toChatRef = \case
       (Just contactId, Nothing) -> Right $ ChatRef CTDirect contactId
       (Nothing, Just groupId) -> Right $ ChatRef CTGroup groupId
-      (_, _) -> Left $ SEBadChatItem itemId Nothing
+      (_, _) -> Left $ SEBadChatItem itemId
 
 getAChatItem :: DB.Connection -> VersionRange -> User -> ChatRef -> ChatItemId -> ExceptT StoreError IO AChatItem
 getAChatItem db vr user chatRef itemId = case chatRef of
@@ -2232,6 +2145,11 @@ getChatItemVersions db itemId = do
       let formattedText = parseMaybeMarkdownList $ msgContentText msgContent
        in ChatItemVersion {chatItemVersionId, msgContent, formattedText, itemVersionTs, createdAt}
 
+getDirectChatReactions_ :: DB.Connection -> Contact -> Chat 'CTDirect -> IO (Chat 'CTDirect)
+getDirectChatReactions_ db ct c@Chat {chatItems} = do
+  chatItems' <- mapM (directCIWithReactions db ct) chatItems
+  pure c {chatItems = chatItems'}
+
 directCIWithReactions :: DB.Connection -> Contact -> CChatItem 'CTDirect -> IO (CChatItem 'CTDirect)
 directCIWithReactions db ct cci@(CChatItem md ci@ChatItem {meta = CIMeta {itemSharedMsgId}}) = case itemSharedMsgId of
   Just sharedMsgId -> do

src/Simplex/Chat/Store/Migrations.hs

@@ -98,7 +98,6 @@ import Simplex.Chat.Migrations.M20240102_note_folders
 import Simplex.Chat.Migrations.M20240104_members_profile_update
 import Simplex.Chat.Migrations.M20240115_block_member_for_all
 import Simplex.Chat.Migrations.M20240122_indexes
-import Simplex.Chat.Migrations.M20240214_redirect_file_id
 import Simplex.Messaging.Agent.Store.SQLite.Migrations (Migration (..))
 
 schemaMigrations :: [(String, Query, Maybe Query)]
@@ -196,8 +195,7 @@ schemaMigrations =
     ("20240102_note_folders", m20240102_note_folders, Just down_m20240102_note_folders),
     ("20240104_members_profile_update", m20240104_members_profile_update, Just down_m20240104_members_profile_update),
     ("20240115_block_member_for_all", m20240115_block_member_for_all, Just down_m20240115_block_member_for_all),
-    ("20240122_indexes", m20240122_indexes, Just down_m20240122_indexes),
+    ("20240122_indexes", m20240122_indexes, Just down_m20240122_indexes)
-    ("20240214_redirect_file_id", m20240214_redirect_file_id, Just down_m20240214_redirect_file_id)
   ]
 
 -- | The list of migrations in ascending order by date

src/Simplex/Chat/Store/Profiles.hs

@@ -388,6 +388,7 @@ deleteUserAddress db user@User {userId} = do
           JOIN user_contact_links uc USING (user_contact_link_id)
           WHERE uc.user_id = :user_id AND uc.local_display_name = '' AND uc.group_id IS NULL
         )
+        AND local_display_name NOT IN (SELECT local_display_name FROM users)
     |]
     [":user_id" := userId]
   DB.executeNamed

src/Simplex/Chat/Store/Shared.hs

@@ -92,12 +92,11 @@ data StoreError
   | SEUniqueID
   | SELargeMsg
   | SEInternalError {message :: String}
-  | SEBadChatItem {itemId :: ChatItemId, itemTs :: Maybe ChatItemTs}
+  | SEBadChatItem {itemId :: ChatItemId}
   | SEChatItemNotFound {itemId :: ChatItemId}
   | SEChatItemNotFoundByText {text :: Text}
   | SEChatItemSharedMsgIdNotFound {sharedMsgId :: SharedMsgId}
   | SEChatItemNotFoundByFileId {fileId :: FileTransferId}
-  | SEChatItemNotFoundByContactId {contactId :: ContactId}
   | SEChatItemNotFoundByGroupId {groupId :: GroupId}
   | SEProfileNotFound {profileId :: Int64}
   | SEDuplicateGroupLink {groupInfo :: GroupInfo}

src/Simplex/Chat/Types.hs

@@ -1210,7 +1210,6 @@ data FileTransfer
 data FileTransferMeta = FileTransferMeta
   { fileId :: FileTransferId,
     xftpSndFile :: Maybe XFTPSndFile,
-    xftpRedirectFor :: Maybe FileTransferId,
     fileName :: String,
     filePath :: String,
     fileSize :: Integer,

src/Simplex/Chat/View.hs

@@ -198,24 +198,17 @@ responseToView hu@(currentRH, user_) ChatConfig {logLevel, showReactions, showRe
   CRGroupMemberUpdated {} -> []
   CRContactsMerged u intoCt mergedCt ct' -> ttyUser u $ viewContactsMerged intoCt mergedCt ct'
   CRReceivedContactRequest u UserContactRequest {localDisplayName = c, profile} -> ttyUser u $ viewReceivedContactRequest c profile
-  CRRcvStandaloneFileCreated u ft -> ttyUser u $ receivingFileStandalone "started" ft
   CRRcvFileStart u ci -> ttyUser u $ receivingFile_' hu testView "started" ci
   CRRcvFileComplete u ci -> ttyUser u $ receivingFile_' hu testView "completed" ci
-  CRRcvStandaloneFileComplete u _ ft -> ttyUser u $ receivingFileStandalone "completed" ft
   CRRcvFileSndCancelled u _ ft -> ttyUser u $ viewRcvFileSndCancelled ft
-  CRRcvFileError u (Just ci) e _ -> ttyUser u $ receivingFile_' hu testView "error" ci <> [sShow e]
+  CRRcvFileError u ci e -> ttyUser u $ receivingFile_' hu testView "error" ci <> [sShow e]
-  CRRcvFileError u Nothing e ft -> ttyUser u $ receivingFileStandalone "error" ft <> [sShow e]
   CRSndFileStart u _ ft -> ttyUser u $ sendingFile_ "started" ft
   CRSndFileComplete u _ ft -> ttyUser u $ sendingFile_ "completed" ft
-  CRSndStandaloneFileCreated u ft -> ttyUser u $ uploadingFileStandalone "started" ft
   CRSndFileStartXFTP {} -> []
   CRSndFileProgressXFTP {} -> []
-  CRSndFileRedirectStartXFTP u ft ftRedirect -> ttyUser u $ standaloneUploadRedirect ft ftRedirect
-  CRSndStandaloneFileComplete u ft uris -> ttyUser u $ standaloneUploadComplete ft uris
   CRSndFileCompleteXFTP u ci _ -> ttyUser u $ uploadingFile "completed" ci
   CRSndFileCancelledXFTP {} -> []
-  CRSndFileError u Nothing ft -> ttyUser u $ uploadingFileStandalone "error" ft
+  CRSndFileError u ci -> ttyUser u $ uploadingFile "error" ci
-  CRSndFileError u (Just ci) _ -> ttyUser u $ uploadingFile "error" ci
   CRSndFileRcvCancelled u _ ft@SndFileTransfer {recipientDisplayName = c} ->
     ttyUser u [ttyContact c <> " cancelled receiving " <> sndFile ft]
   CRContactConnecting u _ -> ttyUser u []
@@ -1565,26 +1558,11 @@ sendingFile_ status ft@SndFileTransfer {recipientDisplayName = c} =
   [status <> " sending " <> sndFile ft <> " to " <> ttyContact c]
 
 uploadingFile :: StyledString -> AChatItem -> [StyledString]
-uploadingFile status = \case
+uploadingFile status (AChatItem _ _ (DirectChat Contact {localDisplayName = c}) ChatItem {file = Just CIFile {fileId, fileName}, chatDir = CIDirectSnd}) =
-  AChatItem _ _ (DirectChat Contact {localDisplayName = c}) ChatItem {file = Just CIFile {fileId, fileName}, chatDir = CIDirectSnd} ->
+  [status <> " uploading " <> fileTransferStr fileId fileName <> " for " <> ttyContact c]
-    [status <> " uploading " <> fileTransferStr fileId fileName <> " for " <> ttyContact c]
+uploadingFile status (AChatItem _ _ (GroupChat g) ChatItem {file = Just CIFile {fileId, fileName}, chatDir = CIGroupSnd}) =
-  AChatItem _ _ (GroupChat g) ChatItem {file = Just CIFile {fileId, fileName}, chatDir = CIGroupSnd} ->
+  [status <> " uploading " <> fileTransferStr fileId fileName <> " for " <> ttyGroup' g]
-    [status <> " uploading " <> fileTransferStr fileId fileName <> " for " <> ttyGroup' g]
+uploadingFile status _ = [status <> " uploading file"] -- shouldn't happen
-  _ -> [status <> " uploading file"]
-
-uploadingFileStandalone :: StyledString -> FileTransferMeta -> [StyledString]
-uploadingFileStandalone status FileTransferMeta {fileId, fileName} = [status <> " standalone uploading " <> fileTransferStr fileId fileName]
-
-standaloneUploadRedirect :: FileTransferMeta -> FileTransferMeta -> [StyledString]
-standaloneUploadRedirect FileTransferMeta {fileId, fileName} FileTransferMeta {fileId = redirectId} =
-  [fileTransferStr fileId fileName <> " uploaded, preparing redirect file " <> sShow redirectId]
-
-standaloneUploadComplete :: FileTransferMeta -> [Text] -> [StyledString]
-standaloneUploadComplete FileTransferMeta {fileId, fileName} = \case
-  [] -> [fileTransferStr fileId fileName <> " upload complete."]
-  uris ->
-    fileTransferStr fileId fileName <> " upload complete. download with:"
-    : map plain uris
 
 sndFile :: SndFileTransfer -> StyledString
 sndFile SndFileTransfer {fileId, fileName} = fileTransferStr fileId fileName
@@ -1630,11 +1608,7 @@ receivingFile_' hu testView status (AChatItem _ _ chat ChatItem {file = Just CIF
               highlight ("/get remote file " <> show rhId <> " " <> LB.unpack (J.encode RemoteFile {userId, fileId, sent = False, fileSource = f}))
             ]
       _ -> []
-receivingFile_' _ _ status _ = [plain status <> " receiving file"]
+receivingFile_' _ _ status _ = [plain status <> " receiving file"] -- shouldn't happen
-
-receivingFileStandalone :: String -> RcvFileTransfer -> [StyledString]
-receivingFileStandalone status RcvFileTransfer {fileId, fileInvitation = FileInvitation {fileName}} =
-  [plain status <> " standalone receiving " <> fileTransferStr fileId fileName]
 
 viewLocalFile :: StyledString -> CIFile d -> CurrentTime -> TimeZone -> CIMeta c d -> [StyledString]
 viewLocalFile to CIFile {fileId, fileSource} ts tz = case fileSource of
@@ -1653,7 +1627,7 @@ fileFrom _ _ = ""
 
 receivingFile_ :: StyledString -> RcvFileTransfer -> [StyledString]
 receivingFile_ status ft@RcvFileTransfer {senderDisplayName = c} =
-  [status <> " receiving " <> rcvFile ft <> if c == "" then "" else " from " <> ttyContact c]
+  [status <> " receiving " <> rcvFile ft <> " from " <> ttyContact c]
 
 rcvFile :: RcvFileTransfer -> StyledString
 rcvFile RcvFileTransfer {fileId, fileInvitation = FileInvitation {fileName}} = fileTransferStr fileId fileName

tests/ChatTests/Files.hs

@@ -9,7 +9,6 @@ import ChatClient
 import ChatTests.Utils
 import Control.Concurrent (threadDelay)
 import Control.Concurrent.Async (concurrently_)
-import Control.Logger.Simple
 import qualified Data.Aeson as J
 import qualified Data.ByteString.Char8 as B
 import qualified Data.ByteString.Lazy.Char8 as LB
@@ -78,11 +77,6 @@ chatFileTests = do
     it "cancel receiving file, repeat receive" testXFTPCancelRcvRepeat
     it "should accept file automatically with CLI option" testAutoAcceptFile
     it "should prohibit file transfers in groups based on preference" testProhibitFiles
-  describe "file transfer over XFTP without chat items" $ do
-    it "send and receive small standalone file" testXFTPStandaloneSmall
-    it "send and receive large standalone file" testXFTPStandaloneLarge
-    xit "removes sent file from server" testXFTPStandaloneCancelSnd -- no error shown in tests
-    it "removes received temporary files" testXFTPStandaloneCancelRcv
 
 runTestFileTransfer :: HasCallStack => TestCC -> TestCC -> IO ()
 runTestFileTransfer alice bob = do
@@ -1551,116 +1545,6 @@ testProhibitFiles =
   where
     cfg = testCfg {xftpFileConfig = Just $ XFTPFileConfig {minFileSize = 0}, tempDir = Just "./tests/tmp"}
 
-testXFTPStandaloneSmall :: HasCallStack => FilePath -> IO ()
-testXFTPStandaloneSmall = testChat2 aliceProfile aliceDesktopProfile $ \src dst -> do
-  withXFTPServer $ do
-    logNote "sending"
-    src ##> "/_upload 1 ./tests/fixtures/test.jpg"
-    src <## "started standalone uploading file 1 (test.jpg)"
-    -- silent progress events
-    threadDelay 250000
-    src <## "file 1 (test.jpg) upload complete. download with:"
-    -- file description fits, enjoy the direct URIs
-    _uri1 <- getTermLine src
-    _uri2 <- getTermLine src
-    uri3 <- getTermLine src
-    _uri4 <- getTermLine src
-
-    logNote "receiving"
-    let dstFile = "./tests/tmp/test.jpg"
-    dst ##> ("/_download 1 " <> uri3 <> " " <> dstFile)
-    dst <## "started standalone receiving file 1 (test.jpg)"
-    -- silent progress events
-    threadDelay 250000
-    dst <## "completed standalone receiving file 1 (test.jpg)"
-    srcBody <- B.readFile "./tests/fixtures/test.jpg"
-    B.readFile dstFile `shouldReturn` srcBody
-
-testXFTPStandaloneLarge :: HasCallStack => FilePath -> IO ()
-testXFTPStandaloneLarge = testChat2 aliceProfile aliceDesktopProfile $ \src dst -> do
-  withXFTPServer $ do
-    xftpCLI ["rand", "./tests/tmp/testfile.in", "17mb"] `shouldReturn` ["File created: " <> "./tests/tmp/testfile.in"]
-
-    logNote "sending"
-    src ##> "/_upload 1 ./tests/tmp/testfile.in"
-    src <## "started standalone uploading file 1 (testfile.in)"
-    -- silent progress events
-    threadDelay 250000
-    src <## "file 1 (testfile.in) uploaded, preparing redirect file 2"
-    src <## "file 1 (testfile.in) upload complete. download with:"
-    uri <- getTermLine src
-    _uri2 <- getTermLine src
-    _uri3 <- getTermLine src
-    _uri4 <- getTermLine src
-
-    logNote "receiving"
-    let dstFile = "./tests/tmp/testfile.out"
-    dst ##> ("/_download 1 " <> uri <> " " <> dstFile)
-    dst <## "started standalone receiving file 1 (testfile.out)"
-    -- silent progress events
-    threadDelay 250000
-    dst <## "completed standalone receiving file 1 (testfile.out)"
-    srcBody <- B.readFile "./tests/tmp/testfile.in"
-    B.readFile dstFile `shouldReturn` srcBody
-
-testXFTPStandaloneCancelSnd :: HasCallStack => FilePath -> IO ()
-testXFTPStandaloneCancelSnd = testChat2 aliceProfile aliceDesktopProfile $ \src dst -> do
-  withXFTPServer $ do
-    xftpCLI ["rand", "./tests/tmp/testfile.in", "17mb"] `shouldReturn` ["File created: " <> "./tests/tmp/testfile.in"]
-
-    logNote "sending"
-    src ##> "/_upload 1 ./tests/tmp/testfile.in"
-    src <## "started standalone uploading file 1 (testfile.in)"
-    -- silent progress events
-    threadDelay 250000
-    src <## "file 1 (testfile.in) uploaded, preparing redirect file 2"
-    src <## "file 1 (testfile.in) upload complete. download with:"
-    uri <- getTermLine src
-    _uri2 <- getTermLine src
-    _uri3 <- getTermLine src
-    _uri4 <- getTermLine src
-
-    logNote "cancelling"
-    src ##> "/fc 1"
-    src <## "cancelled sending file 1 (testfile.in)"
-    threadDelay 1000000
-
-    logNote "trying to receive cancelled"
-    dst ##> ("/_download 1 " <> uri <> " " <> "./tests/tmp/should.not.extist")
-    dst <## "started standalone receiving file 1 (should.not.extist)"
-    threadDelay 100000
-    logWarn "no error?"
-    dst <## "error receiving file 1 (should.not.extist)"
-    dst <## "INTERNAL {internalErr = \"XFTP {xftpErr = AUTH}\"}"
-
-testXFTPStandaloneCancelRcv :: HasCallStack => FilePath -> IO ()
-testXFTPStandaloneCancelRcv = testChat2 aliceProfile aliceDesktopProfile $ \src dst -> do
-  withXFTPServer $ do
-    xftpCLI ["rand", "./tests/tmp/testfile.in", "17mb"] `shouldReturn` ["File created: " <> "./tests/tmp/testfile.in"]
-
-    logNote "sending"
-    src ##> "/_upload 1 ./tests/tmp/testfile.in"
-    src <## "started standalone uploading file 1 (testfile.in)"
-    -- silent progress events
-    threadDelay 250000
-    src <## "file 1 (testfile.in) uploaded, preparing redirect file 2"
-    src <## "file 1 (testfile.in) upload complete. download with:"
-    uri <- getTermLine src
-    _uri2 <- getTermLine src
-    _uri3 <- getTermLine src
-    _uri4 <- getTermLine src
-
-    logNote "receiving"
-    let dstFile = "./tests/tmp/testfile.out"
-    dst ##> ("/_download 1 " <> uri <> " " <> dstFile)
-    dst <## "started standalone receiving file 1 (testfile.out)"
-    threadDelay 25000 -- give workers some time to avoid internal errors from starting tasks
-    logNote "cancelling"
-    dst ##> "/fc 1"
-    dst <## "cancelled receiving file 1 (testfile.out)"
-    threadDelay 25000
-    doesFileExist dstFile `shouldReturn` False
-
 startFileTransfer :: HasCallStack => TestCC -> TestCC -> IO ()
 startFileTransfer alice bob =
   startFileTransfer' alice bob "test.jpg" "136.5 KiB / 139737 bytes"

tests/ChatTests/Local.hs

@@ -152,7 +152,7 @@ testFiles tmp = withNewTestChat tmp "alice" aliceProfile $ \alice -> do
 
   alice ##> "/clear *"
   alice ##> "/fs 1"
-  alice <## "file 1 not found"
+  alice <## "chat db error: SEChatItemNotFoundByFileId {fileId = 1}"
   alice ##> "/tail"
   doesFileExist stored `shouldReturn` False