Organization of client-server data synchronization for a mobile application

Lecture



There is a client and server.
Now each stores its data, which is entered separately to the client and separately to the server.
Data can be created / modified / deleted on the client and on the server.

Changes can be (in the target model) to make both on the client and on the server (from under one account, or admin).
It is necessary to describe how to synchronize this data correctly.

To more clearly describe what is required, take for example the application instagram.
I made a photo on Instagram - it is visible to me, then on the server and at my friends.
Someone wrote a comment through the web version and it was reflected in my friends and me.
Someone liked himself - and again everything was updated along the chain - the user who was liking, the server, me and my friends.
I wiped my comment from someone and it got stuck with everyone who could see it.

How to implement a similar synchronization scheme ?
What is needed for this?

Architecturally possible two solutions.

1. API outside processing mobile application requests.
2. Local storage of the settings file (sqlite, etc) and its upload / download to the cloud (dropbox, icloud, yadisk, etc)

The work of the API can be done according to the following scheme.
The local structure stores two identifiers, one completely local, the other - the one that came from the server after creation.

The mechanism is: did something, is there an Internet connection?
YES - send to server - get a successful response?
Yes - added to the local database with the id that came from the server.
no - we write what the error is to the log, we apologize to the user (although the scheme is different, usually everything is ok, and not ok only if the server has fallen)
NO - we write to the local database, with an empty server ID

network connection appears
- we send local objects in portions (if they were not deleted before the appearance of the Internet) to get the required server ID
- we request the list of deleted from the server (perhaps under this account on another device something was deleted) we delete local objects

As soon as the server receives the data, it reports on the socket to all users who are online and in some way connected with the user's content, what it did.

You have only two synchronization methods:
- http request that is sent by the client to download / send data;
- PUSH notification that the server sends to the client;

Example:
1. Customer A took a photo.
2. Client A sent a photo to the server.
3. Server received photo
4. The server sent PUSH notifications to clients B and C with the information "there is a new photo from client A"
5. Customers B and C have received notifications.
5.1 If the user (person) is in the photo tape, the client makes a request to the server, uploads the photo and displays it.
5.2 If the user is, for example, on the settings screen, the client displays a notification (toast) "there is a new photo from client A"
Implementation example

Method 1 One-way synchronization of state changes

Organization of client-server data synchronization for a mobile application

Method 2 One-way data synchronization

Organization of client-server data synchronization for a mobile application
Implementation notes
  •   passed parameters (fields)
      1.data- line in json format
     
      
       Description:
      
     1. data in the data field is transmitted in a string in json format 
      In them we transfer data which needs to be synchronized.
      It uses:
     
       a) the unique local creation time (change) "ut" (each entry)
       b) and a unique local id (for each entity) (id must be unique, have a - sign).
        c) transmission without a minus sign for the id, if id global. 
  • example
  •   data =
      [
         {
             "cmd": 1,
             "ut": 134256789088,
             "tb": "product",
             "dt": [-6341,134,13,12,12312,12]
          },
         {"cmd": 2,
            "ut": 4578678789876,
            "tb": "product",
            "dt": [1,12,162,124,142,12,12, -341, 126,12,122,122]
         },
    	 {"cmd": 3,
            "ut": 45678908765,
            "tb": "product",
            "dt": [1]
         }
    
     ]
     
      Where 
     
        cmd - command (1,2,3) for details see below.
        ut - unictstaym (in seconds) of each record on the device
       (the time of the last local work with this object)
        dt- data array without field names 
        tb - table name for synchronization
        see values ​​below
     
     
     
     
      2. transfer no more than N records in one request, if there are more than N records, then transfer first the read records to the record, then everything else in portions (page by page) 
  • Comments

    1. if the device received a negative response from the server (other than 200 and after a retransmission without synchronization)
    or the device does not have access to the Internet then write all data to the local database

    after the appearance of the Internet or in the presence of such data to determine the difference previously synchronized data
    and new data collected offline

    the result of these comparisons should be the objects and the three operations on them add, update, delete
    necessary for the bases to be the same

    2. send the following data
    --with id <0 (application stores all records created offline with global id = 0,
    but you need to transfer with a local ID with a minus sign) - if a new object is used, it can be used in the associated data
    (after sending the command for synchronization and if successful, the application should update the global ids for server values)
    - with a sign of update (the application must have an update flag for each entry of synchronized entities)
    This flag can be reset only after sending the command to synchronize and if it was successful.
    --with an id in the deletion table (the application must store all id records and the name of entities that are deleted offline)
    after sending the command to synchronize and if it is successful, the application clears the list of deleted entries



    3. send kommandu on synchronization
    if successful, we load all the data or load the modified data and delete the sequential data in the application
    (at this moment a deferred recording is formed for synchronization in linked accounts via PUSH)

    4. Customer stores
    1. new entries must have id <0 and equal to unique local number
    2. Remote records should be stored in the deletes table.
    3. It is advisable to mark updated records in offline mode with a sign of updates.

    5. Names of operations
    ins = 1
    upd = 2
    del = 3


    6.Name of entities and fields available for insertion or update (if you need to delete a record, you need only an id):
    It is necessary to transfer the data in this order without the name of the fields;

    entity1 [id field id .....]
    Entity2 [id field id .....]
    Entity3 [id field id .....]

    so synchronization from the device to the server consists of such actions
    1. transfer data (in this case, a complex of checks on the server is released)

    2. save changes (changes are made sorted by unixtime)
    and changes are made (synchronization via push) in a queue for related
    with this account devices (accounts)

    3. if the server has an entry with unixtime more than the device,
    then changing or deleting this entry is ignored and such entry
    the device needs to be downloaded from the server already

    4. after transfer and fixation of data on the server you need to get
    data (from p3) from the server and transfer to the local database of the device
    5. because there is no backward synchronization, it is desirable to perform a full data load at the end
    (to update the id), this is necessary if you don’t deliver, or push back with a request for reverse synchronization, or the data transferred is not taken into account
    to push

    notes
    1. negative local id is used in three cases
    if new record is added - always
    (must be transmitted before its use in other records)
    if the record created locally is updated or deleted, at the same time, such a record is not transmitted in case of synchronization using method 2
    if the existing record is updated but with indication in the field to the local record

Comments


To leave a comment
If you have any suggestion, idea, thanks or comment, feel free to write. We really value feedback and are glad to hear your opinion.
To reply

Databases, knowledge and data warehousing. Big data, DBMS and SQL and noSQL

Terms: Databases, knowledge and data warehousing. Big data, DBMS and SQL and noSQL