+ The Oktave Forum » Technical » Engineering » Data Management (Moderator: mandar)
|-+ Spiralling joins
Username:
Password:

Pages: [1]
Topic Tools  
Read July 08, 2008, 03:15:38 pm #0
sri

Spiralling joins

I have a problem in a database application that I am designing. Looking for some good ideas.

In this application, I have entities organized in the form of a tree. Each entity has several children, which in turn may have several children themselves and so on. This relationship is represented by a foreign key to the parent, from the entity table to itself. Of course, indexes are maintained.

But the problem is this: Given an entity, I need to pre-read the entity and all its descendants till a certain depth. Here is where the problem comes in. At every level, I have to compute a join of this table with itself to compute the next set of children. And of course, the rate at which the number of children grow is multiplicative for each iteration.

Because so many joins have to be computed, the entire process becomes very slow. So slow that for a realistic dataset, the server OS starts thrashing.

Is there a way to compute these spiralling joins fast without materializing the view? The database is fairly prone to updates, so materialization is not a good idea.

Any ideas welcome..
-Sri
« Last Edit: July 08, 2008, 03:51:54 pm by sri »
Offline  
Read July 08, 2008, 05:05:26 pm #1
sids

Re: Spiralling joins

This sounds like a real world use-case for a Graph Database!

When you say "the database is fairly prone to updates", do you mean that the tree itself is prone to updates or only other attributes of the entities are? If the tree is not prone to updates, then maybe you can have a pointer to all the grandparents in addition to the parent itself? This would require an additional lookup when adding a new node and would need an update to all the child nodes if a node's parent is changed. But if these operations are fairly infrequent, this approach might work. Of course, this might not be usable if you need attributes other than the primary key from the parents (and would further complicate it otherwise too if the primary key can be updated).


http://www.grok.in/
"Ignorance killed the cat, curiosity was framed."
Offline  
Read July 09, 2008, 02:33:39 am #2
sri

Re: Spiralling joins

When you say "the database is fairly prone to updates", do you mean that the tree itself is prone to updates or only other attributes of the entities are? If the tree is not prone to updates, then maybe you can have a pointer to all the grandparents in addition to the parent itself? This would require an additional lookup when adding a new node and would need an update to all the child nodes if a node's parent is changed. But if these operations are fairly infrequent, this approach might work. Of course, this might not be usable if you need attributes other than the primary key from the parents (and would further complicate it otherwise too if the primary key can be updated).

This was what we had done in our graph database model by building an index structure (LWI) that indexed all possible paths in the database.

But no, in this case this does not work Sad

The structure is also prone to updates as much as the data itself. And the entities are not necessarily "labeled" where the number of labels is far lesser than the number of entities, so that they can be somehow used..
Offline  
Read July 09, 2008, 06:58:02 pm #3
sids

Re: Spiralling joins

I can't think of any "magic bullet" solution to this problem. But using a good in-memory cache might mitigate it to a large extent. For example, you can pre-calculate the top few levels (say 3) of the tree and always keep it cached in memory. You could also cache the tree calculated below the top levels but have an upper limit on the number of paths you will cache and use some replacement policy (LRU, LFU etc.) to discard old or unused paths. memcached is an open-source distributed memory object caching system which is widely deployed in similar situations.

Hope this helps.


http://www.grok.in/
"Ignorance killed the cat, curiosity was framed."
Offline  
Pages: [1]
Jump to: