-
-
Save a2ndrade/08bb7417b901e904d3b4 to your computer and use it in GitHub Desktop.
Atomic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| / c: count | ht: historical table | st: snapshot table | rt: realtime table | tx: transaction | |
| createRandomHdbFrom:{[c;u]flip`e`v`t!(c?c*4;u@c?count u;1+til c)} | |
| createRandomHdb:createRandomHdbFrom[;"abcdefghijklmnopqrstuvwxyz"] | |
| appendToHdbFrom:{[ht;c;u]ht upsert flip(c?100000*count u;c?u;(1+$[0~count lastTx:select [-1] t from ht;0;first lastTx@`t]))} | |
| appendToHdb:appendToHdbFrom[;;"abcdefghijklmnopqrstuvwxyz"] | |
| createSnapshotTable:{[ht;tx]tx:$[-1~tx;0Wj;tx];s:`v xasc select e,v from ht where t <= tx;w:last each value group s@`e;`e xkey s@w} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| \c 30 300 | |
| / KDB built-in types | |
| / default values | |
| STORAGE_VALUES:(0b;0Ng;0x00;0h;0i;0;0e;0.0;" ";`;0Np;2000.01m;2000.01.01;2000.01.01T00:00:00.000;0Nn;00:00;00:00:00;00:00:00.000;0) | |
| STORAGE_VALUES,:enlist each STORAGE_VALUES | |
| / char aliases | |
| STORAGE_CHARS:"bgxhijefcspmdznuvtr" / scalar char aliases | |
| STORAGE_CHARS,:upper STORAGE_CHARS / list char aliases | |
| / friendly names | |
| STORAGE_NAMES:`boolean`guid`byte`short`int`long`real`float`char`symbol`timestamp`month`date`datetime`timespan`minute`second`time`ref | |
| STORAGE_NAMES,:{`$,[;"Vector"] each string x} STORAGE_NAMES | |
| STORAGE_TYPES:type each STORAGE_VALUES | |
| / Historical DB utility functions | |
| getHdbTableHandle:{[a]`$,[;"/"] ":",string 0x0 sv md5 string a} | |
| isPersistedHdbTable:{[th]11h~type key th} | |
| isPersistedAttribute:{[a]isPersistedHdbTable getHdbTableHandle a} | |
| createHdbTable:{[a]$[isPersistedHdbTable th:getHdbTableHandle a;th;.[th;();:;([]e:();v:())]]} | |
| appendToHdbSystemTable:{[systemAttributes;a;attrValues] | |
| th:getHdbTableHandle[a]; | |
| rows:([]e:til count systemAttributes; v:attrValues); | |
| upsert[th;rows]} | |
| initHdb:{[nextEntityId;nextTxId] | |
| / `A` is an in-memory table that contains the definition of the core built-in attributes in the database | |
| attrEntityId:-1; / negative numbers are temp ids used to specify relationships between entities before those entities are persisted | |
| A:flip`entityId`entityDt`attrName`attrType`attrIndex`attrFullText`entityDoc! | |
| flip( / Attribute definition. Note entityId is a virtual attributes i.e. have no explicit storage representation | |
| (0N;attrEntityId;`entityDt;"r";1b;0b;"This entity's datatype. Every object in the system gets its id from this table. Every object has a type"); | |
| (0N;attrEntityId;`attrName;"s";1b;0b;"Unique identifier among all attributes. This also controls the table name on disk"); | |
| (0N;attrEntityId;`attrType;"h";1b;0b;"kdb+ storage type char; see http://code.kx.com/wiki/Reference/Datatypes"); | |
| (0N;attrEntityId;`attrIndex;"b";1b;0b;"1b if designer expects to filter or sort on this attribute; 0b if not. Defaults to 1b"); | |
| (0N;attrEntityId;`attrFullText;"b";1b;0b;"1b if attribute value should be indexed for full-text search; 0b if not. Defaults to 1b"); | |
| (0N;attrEntityId;`entityDoc;"C";1b;0b;"Documentation string"); | |
| / First-class transactions | |
| (0N;attrEntityId;`txTimestamp;"p";1b;0b;"The timestamp associated with this transaction."); | |
| / Datatype meta-model | |
| (0N;attrEntityId;`dtNamespace;"s";1b;0b;"Datatype's namespace"); | |
| (0N;attrEntityId;`dtName;"s";1b;0b;"Datatype's local name. Names within a given namespace are unique"); | |
| (0N;attrEntityId;`dtParent;"r";1b;0b;"Datatype's parent datatype. Only populated in datatype hierarchies"); | |
| (0N;attrEntityId;`dtList;"r";1b;0b;"Datatype's list datatype"); | |
| (0N;attrEntityId;`dtComponent;"r";1b;0b;"Datatype's component datatype"); | |
| (0N;attrEntityId;`dtFields;"R";1b;0b;"Datatype's fields"); | |
| / Security model | |
| (0N;attrEntityId;`userName;"s";1b;0b;"User"); | |
| (0N;attrEntityId;`groupId;"r";1b;0b;"Group"); | |
| (0N;attrEntityId;`roleName;"s";1b;0b;"Role name. Unique across the system"); | |
| (0N;attrEntityId;`roleMapEntryRole;"r";1b;0b;"Reference to a RoleMap entity"); | |
| (0N;attrEntityId;`roleMapEntryUsers;"R";1b;0b;"References to User entities"); | |
| (0N;attrEntityId;`roleMapEntryGroups;"R";1b;0b;"References to Group entities"); | |
| (0N;attrEntityId;`roleMapEntries;"R";1b;0b;"References to RoleMapEntry entities")); | |
| /A[`attrType]:STORAGE_TYPES@STORAGE_CHARS?A[`attrType]; / overrides type chars with type names | |
| / Core Datatypes: Datatype + Attribute + User + Group + Role + RoleMapEntry + RoleMap | |
| / This is an in-memory table that provides fast access to all datatype definitions in the database | |
| namespace:`$"http://www.appian.com/ae/types/2009"; | |
| dtEntityId:-2; | |
| D:flip`entityId`entityDt`dtNamespace`dtName`dtParent`dtFields`entityDoc! | |
| flip((dtEntityId;dtEntityId;namespace;`Datatype;0N;`entityDt`dtNamespace`dtName`dtParent`dtList`dtComponent`dtFields;"The datatype for datatypes"); | |
| (attrEntityId;dtEntityId;namespace;`Attribute;0N;`entityDt`attrName`attrType`attrIndex`attrFullText;"The attribute datatype"); | |
| (0N;dtEntityId;namespace;`User;0N;enlist`userName;"The user datatype"); | |
| (0N;dtEntityId;namespace;`Group;0N;enlist`groupId;"The group datatype"); | |
| (0N;dtEntityId;namespace;`Role;0N;enlist`roleName;"The role datatype"); | |
| (0N;dtEntityId;namespace;`RoleMapEntry;0N;`roleMapEntryRole`roleMapEntryUsers`roleMapEntryGroups;"The rolemap entry datatype"); | |
| (0N;dtEntityId;namespace;`RoleMap;0N;enlist`roleMapEntries;"The rolemap datatype")); | |
| / Initialize in-memory tables | |
| tmpEIds:(A`entityId),D`entityId; | |
| eIds:value eIdsMapping::allocateNewEntityIds[nextEntityId;tmpEIds]; / temporary ids TO persisted ids table | |
| / 1) assign first X ids from eIds + 2) override temp datatype entity ids with new allocated ids | |
| A:update entityId:(count A)#eIds,entityDt:eIdsMapping[entityDt] from A; | |
| D:update entityId:(count A)_eIds,entityDt:eIdsMapping[entityDt] from D; | |
| / overrides friendly attribute names with their primary key in the `A` table | |
| D[`dtFields]:A[`entityId]@(A`attrName)?D`dtFields; / if `A` is an unkeyed table | |
| /D[`dtFields]:{exec entityId from A where attrName in x} each D`dtFields / if `A` is a keyed table | |
| / join table A and D into a single one (data) | |
| data:(count tmpEIds)#(); | |
| data[til count A]:A; | |
| data[(count A) + til count D]:D; | |
| /data[;`entityId]:eIds; | |
| / figure out ALL distinct attribute names being written to | |
| k:(distinct raze key each data) except `entityId; | |
| / mask of where each attribute appears | |
| w:where each (count each data)>/:(key each data)?\:/:k; | |
| / get ids by attribute | |
| e:eIds@w; | |
| / get values by attribute | |
| v:(flip data[;k])@'w; | |
| / create in-memory EV tables keyed by attribute name | |
| T:k!{[e;v;t;w]flip`e`v`t!(e[w];v[w];t)}[e;v;nextTxId] each til count k; | |
| :`eIds`attributes`datatypes`tables!(eIdsMapping;A;D;T)} | |
| allocateNewEntityIds:{[nextEntityId;tmpEntityIds] | |
| eIds:(count tmpEntityIds)#0N; | |
| eIds[where mNull]:nextEntityId+til countNulls:0 +/ mNull:0N=tmpEntityIds; | |
| tmpIdsMapping:tmpIds!nextEntityId+countNulls+til count tmpIds:distinct tmpEntityIds@wNotNull:where not mNull; | |
| eIds[wNotNull]:tmpIdsMapping@tmpEntityIds[wNotNull]; | |
| :tmpEntityIds!eIds} | |
| DB:0N!initHdb[1;1] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment