Molaskes: Paving and Leading the Way

DDGDS stands for "Dynamically Defined Giga-user Data System". It is a massively multiuser database (MMDB) to be used for instance in large social or e-commerce websites. It uses F‌ The GDS File Format (GDS = "Grouped Data String") very efficiently. While media files are stored with their proper file extensions (e.g. *.jpg), the database files of DDGDS are all extensionless. The DDGDS works with GDS-compatible arrays that use number strings as keys (for instance "12") instead of the actual alphanumeric keys (for instance "message_body") used by the programmer, which are kept in a separate file type definition array _types, using special read/write functions that dynamically handle the assignments for the programmer. This significantly saves disk space on the server. Here is a schematic example of a _types definition: ‌ The key names in a block must be unique, as the software will work with a reverse lookup structure. In our example, this would look like the following: ‌ No two key names in the same block must point to the same block-specific index number, which is why the key names must be unique within their block. While you can append anytime new fields to any block, for backward compatibility reasons you should never reorder the items in a block that is already in use, nor delete items from the block. Items that have become obsolete may be named by unique special key names such as in this example (note the two fields "unused1" and "unused2"): ‌ All DDGDS arrays are accessed only indirectly through management functions by the application programmers, never as the raw arrays. We will look at Eas examples (see E‌ →Molaskes.info/Eas) for these functions, assuming having read from disk a user file as defined above. The GDS-decoded array will be stored in the variable userVar. To make the top level "palatable", we use userVarX DDGDS'Get:userVar. The variable userVarX will then be a regular keyed array with perhaps the following contents: The groups here are a space-separated list of item IDs (more about that later), but more important for us is that the friends, which in the format definition above form a sub-block, are only given as a single number. This is because such sub-blocks always imply an unkeyed list/array with each element defined as in the sub-block. The value we got here is the count of the elements, so the number of Lara's friends in this case. Alternatively, you can read out any value directly, for instance via DDGDS'Get:userVar "avatar" which here would return "lara2.jpg". If we now wanted to read the 6th friend's data (mind that indexes start with 0), we would simply call curpal DDGDS'Get:userVar "friends:5", which may set the variable curpal to the following contents: ‌ Writing only works with direct fields, so if Lara changed her avatar picture, we might call DDGDS'Set:userVar "avatar" "lara_beach.jpg", and adding a new friend for her might be done with: ‌ For data safety reasons, writing empty values should not be allowed, and one should rather use a designated function to delete items, which is handled correctly by the function to keep key:index assignments where they are required. Unkeyed list entries however will actually be deleted also in the raw data. Here are some examples for deleting elements: Mind though that deleting unkeyed list items shifts all indexes in the same list that come after it. So in our example, the "friends:35" entry would shift to "friends:34" after deleting "friends:20". All files in DDGDS get an item ID that is the Eas BASE:64 encoded expression of its creation DateTime value. Its uniqueness per directory is ensured by a database locking protocol (lock-by-user—check-user—write—unlock). (Eas BASE:64 differs from most older Base64 implementations as it is fully compatible with any lower-base mathematical notatioons, using 0–9 for the first ten digits (decimals), then uppercase A–Z (hexadecimals and beyond, for values 10–35), then the underscore "_" character for the value 36, then lowercase a–z for the values 37–62, and finally the minus dash "-" charater for the value 63.) If we wanted to load the user data of Lara's friend "Pauly", we would call DDGDS'Load:"user" "bZ3oSu9", which would actually load the file ./user/b/Z/3/o/S/u/9/_, splitting the item ID character by character into a path of subdirectories of not more than 64 different entries each, and using a file simply named by the underscore "_" to store the main data. When Pauly wants to read Lara's latest private message, the server code might for instance call DDGDS'Load:"user" "bZ3oSu9" "pm" "29D3-jX/w3Wkf31" which would actually load the file ./user/b/Z/3/o/S/u/9/pm/29D3-jX/w3Wkf31, with 29D3-jX being Lara's userID and w3Wkf31 being the actual message's ID. After Lara has made changes to her user profile, we can update her data on the server via DDGDS'Save:"user" "29D3-jX" userVar. The same function is used to create new entries on the disk, for instance when sending a new private message. For data privacy reasons, there should be extended functions DDGDS'SaveX and DDGDS'LoadX, where the former obfuscates the file, which just means making it unreadable for humans (for example protecting against admins with direct access to the server data, such as via FTP), and the latter undoing the obfuscation before GDS-parsing it. This is for instance important for private messaging. Admins who maintain the server and might nosily investigate some of the data then would not read for example: Bad news, hun: My STD infection is confirmed. Dang! …‌ but would instead see something like this: 3Iaw_ "oW!s.JRxii74 w:icEFW+ 3qyY -190X?k_ xEw76xzC …. The actual login authentication should use a special directory with subdirectories representing split login names instead of Base64IDs. Usernames would have to be unique when transformed to a simplified authentication form, for instance rendering the user name "René Knödel" as the login name "rene_knoedel", the login authentication data (password authentication seal, and userID) for which would be stored in ./login/r/e/n/e/_/k/n/o/e/d/e/l/_. When the user has entered their username and password, the login authentication check would be performed in a way similar to this: (The randomized login token is used to verify for each access that the user's login cookie is actually valid, the result of a proper login with the correct password.)

Esc Search / Table of Contents

 

Tab ›