WebObjects/EOF/Using EOF/Bulk Operations

Overview

The standard techniques for using EOF can potentially break down when you are dealing with extremely large datasets. The techniques described here are useful for taking advantage of EOF's benefits while not at the same time running out of memory.

There is quite a bit of overlap between this topic and the Memory Management section. It is recommended that you review the general memory management methodologies prior to reading this.

Moving Large Amounts of Data Between Databases

Chuck Hill

Call unlock() then dispose() on the ec, then set the reference to null, then force garbage collection with: http://java.sun.com/docs/books/tutorial/essential/system/garbage.html

You will need to do this often with a large database, maybe more than once per entity. I don't know if that would work while maintaining the object graph.

Pierce T Wetter III

There used to be an EOUtil that could do something like this. Does that still exists?

Otherwise, it depends on how the ER relationship chart works. In our case, we have a set of objects that would more or less "own" all the other objects in the ER tree. So the code would look something like this:

 get master editing context

 fetch copies of all the master objects into the master ec (not that many, so ok)

 for each master object:

    make a local ec
    copy the master object into the local ec (localInstanceOfObject())
     do stuff with master object (i.e. copy them into the sink model)
    toss local ec

Using the local EC is the key.

If you have a large number of master objects enough that you can't load all of them into memory at once, then it gets trickier. One way is to just fetch a list of primary keys using raw rows, then use objectWithPrimaryKeyValue() to build the objects in the local editing context.

If your ER tree is more complex (there is no "owner" object, then YMMV).

EOImportExportApplication from Cassini

Cassini is available at Sourceforge.

Anjo Krank

There is some stuff in Wonder's ERXJDBCUtilities that should be able to copy from one DB to another. I never tried it, but maybe it gives you some hints. There is no real need to use EOs anyway as you just want to copy the data.

David Teran

Hmpf! 'that should be able to copy from one DB to another.' NoNo, Anjo, that is not fair ;-)

In fact it works perfectly -if- the JDBC driver is not buggy. Here is a quick documentation:

OK, you need to use ERExtensions and ERJars. Not a big deal, drop me a line and i'll send a demo app to you.

Then you need this code here, the example takes data from frontbase and saves it to a postgres database:

 NSMutableDictionary sourceDict = new NSMutableDictionary();
 sourceDict.setObjectForKey("", "password");
 sourceDict.setObjectForKey("_SYSTEM", "username");
 sourceDict.setObjectForKey("jdbc:FrontBase://195.135.143.205/merces/user=_system", "url");
 sourceDict.setObjectForKey("com.frontbase.jdbc.FBJDriver", "driver");
 sourceDict.setObjectForKey(Boolean.FALSE, "autoCommit");
 sourceDict.setObjectForKey(Boolean.TRUE, "readOnly");
 
 NSMutableDictionary destDict = sourceDict.mutableClone();
 destDict.setObjectForKey("jdbc:postgresql://195.135.143.205/merces", "url");
 destDict.setObjectForKey("dev", "password");
 destDict.setObjectForKey("dev", "username");
 destDict.setObjectForKey("org.postgresql.Driver", "driver");
 destDict.setObjectForKey(Boolean.FALSE, "autoCommit");
 destDict.setObjectForKey(Boolean.FALSE, "readOnly");
 
 EOModel yourModel  = EOModelGroup.defaultGroup().modelNamed("YourEOModelName");
 ERXJDBCUtilities.copyDatabaseDefinedByEOModelAndConnectionDictionaryToDatabaseWithConnectionDictionary(yourModel,
     sourceDict, destDict);

Thas all!

BulkMover from 360works

BulkMover provides these capabilities, though it is a ProjectBuilder project.