org.archive.util
Class CachedBdbMap<K,V>

java.lang.Object
  java.util.AbstractMap<K,V>
      org.archive.util.CachedBdbMap<K,V>

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.util.concurrent.ConcurrentMap<K,V>, java.util.Map<K,V>, ObjectIdentityCache<K,V>

Deprecated. use ObjectIdentityBdbCache instead

public class CachedBdbMap<K,V>
extends java.util.AbstractMap<K,V>
implements java.util.concurrent.ConcurrentMap<K,V>, ObjectIdentityCache<K,V>, java.io.Serializable, java.io.Closeable

A BDB JE backed hashmap. It extends the normal BDB JE map implementation by holding a cache of soft referenced objects. That is objects are not written to disk until they are not referenced by any other object and therefore can be Garbage Collected.

BDB Java Edition is actually a btree. Flush to disk can be forced by sync().

To ensure that changes/mutations to values in this map are coherent and consistent at the application level, it is assumed that the application level only mutates values that are in this map and does not retain references to values longer than necessary. This allows mappings to be persisted during GC without explicit transactions or write operations.

There are two styles of CachedBdbMap usage:

1. single threaded (or externally synchronized) activity that uses any Map and ConcurrentMap methods available and specifically requires remove().

2. concurrent, high volume, accretive-only activity that uses putIfAbsent(K, V), but not the put(K, V), the replace(K, V, V) or 1 arg remove(java.lang.Object) methods. The concurrent replace(K, V, V) methods can be used if application level logic can rule out surprise unmapping of values between threads. This usage style does not require locking memMap and diskMap together to guarantee cache coherence and consistency.

Both styles rely on an internal expunge operation (or the explicit sync()) to save changes to values. The single threaded case can also use put(K, V) on top of an existing entry as long as no thread retains the previous value instance.

Author:

John Erik Halse, stack, gojomo, paul baclace (conversion to ConcurrentMap)

See Also:

Serialized Form

Nested Class Summary

protected static class	CachedBdbMap.DbEnvironmentEntry Deprecated. Simple structure to keep needed information about a DB Environment.
protected class	CachedBdbMap.LowMemoryCanary Deprecated.

Nested classes/interfaces inherited from class java.util.AbstractMap

java.util.AbstractMap.SimpleEntry<K,V>, java.util.AbstractMap.SimpleImmutableEntry<K,V>

Nested classes/interfaces inherited from interface java.util.Map

java.util.Map.Entry<K,V>

Field Summary

protected java.lang.ref.SoftReference<CachedBdbMap.LowMemoryCanary>	canary Deprecated.
protected com.sleepycat.je.Database	db Deprecated. The BDB JE database used for this instance.
protected com.sleepycat.collections.StoredSortedMap	diskMap Deprecated. The Collection view of the BDB JE database used for this instance.
protected java.util.concurrent.atomic.AtomicInteger	diskMapSize Deprecated. The number of objects in the diskMap StoredMap.
protected java.util.concurrent.ConcurrentHashMap<K,org.archive.util.CachedBdbMap.SoftEntry<V>>	memMap Deprecated. The softreferenced cache of diskMap.
protected static java.lang.reflect.Field	referentField Deprecated. Reference to the Reference#referent Field.
protected java.lang.ref.ReferenceQueue<V>	refQueue Deprecated.

Constructor Summary

CachedBdbMap(java.io.File dbDir, java.lang.String dbName, java.lang.Class<K> keyClass, java.lang.Class<V> valueClass)
Deprecated. A constructor for creating a new CachedBdbMap.

CachedBdbMap(java.lang.String dbName)
Deprecated. Constructor.

Method Summary

void	clear() Deprecated. Note that a call to this method CLOSEs the underlying bdbje.
void	close() Deprecated. close/release any associated resources
boolean	containsKey(java.lang.Object key) Deprecated.
boolean	containsValue(java.lang.Object value) Deprecated.
protected com.sleepycat.collections.StoredSortedMap	createDiskMap(com.sleepycat.je.Database database, com.sleepycat.bind.serial.StoredClassCatalog classCatalog, java.lang.Class keyClass, java.lang.Class valueClass) Deprecated.
java.util.Set<java.util.Map.Entry<K,V>>	entrySet() Deprecated.
protected void	finalize() Deprecated.
V	get(java.lang.Object object) Deprecated. get the object under the given key/name
protected java.lang.String	getDatabaseName() Deprecated.
V	getOrUse(K key, Supplier<V> supplierOrNull) Deprecated. ObjectIdentityCache get-or-atomic-create method.
void	initialize(com.sleepycat.je.Environment env, java.lang.Class<? super V> valueClass, com.sleepycat.bind.serial.StoredClassCatalog classCatalog) Deprecated. Call this method when you have an instance when you used the default constructor or when you have a deserialized instance that you want to reconnect with an extant bdbje environment.
protected void	initializeInstance() Deprecated. Do any instance setup.
protected void	initTransientStats() Deprecated.
java.util.Set<K>	keySet() Deprecated. The keySet of the diskMap is all relevant keys.
protected com.sleepycat.je.Database	openDatabase(com.sleepycat.je.Environment environment, java.lang.String dbName) Deprecated.
V	put(K key, V value) Deprecated. Map.put() implementation.
V	putIfAbsent(K key, V value) Deprecated. A composite putIfAbsent() over memMap and diskMap.
V	remove(java.lang.Object key) Deprecated. Remove mapping for the given key.
boolean	remove(java.lang.Object key, java.lang.Object value) Deprecated. remove item matching both the key and value.
V	replace(K key, V value) Deprecated. Replace entry for key only if currently mapped to some value.
boolean	replace(K key, V oldValue, V newValue) Deprecated. Replace entry for key only if currently mapped to given value.
int	size() Deprecated. count of name-to-object contained
void	sync() Deprecated. Sync in-memory map entries to backing disk store.

Methods inherited from class java.util.AbstractMap

clone, equals, hashCode, isEmpty, putAll, toString, values

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.util.Map

equals, hashCode, isEmpty, putAll, values

Field Detail

db

protected transient com.sleepycat.je.Database db

Deprecated.

The BDB JE database used for this instance.

diskMap

protected transient com.sleepycat.collections.StoredSortedMap diskMap

Deprecated.

The Collection view of the BDB JE database used for this instance.

memMap

protected transient java.util.concurrent.ConcurrentHashMap<K,org.archive.util.CachedBdbMap.SoftEntry<V>> memMap

Deprecated.

The softreferenced cache of diskMap. Policy is: a memMap value is always correct *if present*. The diskMap value is only correct if there is no memMap value (and any phantom memMap values have been persisted.) That is, if memMap has a value, then that value is the most up to date. If memMap does not hold a value, then diskMap has the most up to date value.

For diskMap size monitoring and concurrency support, the following invariant about keys is maintained: If a key is present in memMap, it must also present in diskMap (although the value in diskMap may not be the most recent, as above policy describes).

The key presence invariant and the value coherence policy are maintained by using the "happens after" sense in the Java memory model by transitivity between diskMap and memMap.

The clients of this class "only see one value instance" at a time so that multiple, independent get(k) calls return the same object instance.

Strategy Notes about using ConcurrentMap semantics to implement the desired policy (see method comments for details):

 1. Swap in/First insert: diskMap, 
     then memMap // putIfAbsent() assures atomicity.
 2. Value mutation: only one value instance per key in memMap 
     is maintained so identity compare of value works and all clients 
     operate on the same object.
     This implies that methods which change the instance 
     (replace() methods) are not compatible with this design and
     should only be used if instance of CachedBdbMap is used 
     by a single thread.
     Because content of referent is mutated, mapping is not changed;
    2.1. BDB JE operation assumption: calling get(k) twice returns
     values that are equals(), but not necessarily == by identity
     ( diskMap.get(k).equals(diskMap.get(k)) is true, but 
      diskMap.get(k) != diskMap.get(k) (might be, but not guaranteed)).
 3. Swap out/flush: diskMap update, then memMap remove: 
     diskMap.put(k,v2); // exclusive only if performed when 
     processing ref queue during expunge;
     Avoid expunge races with a countdown latch in SoftEntry;
     memMap.remove(k,v2); if memMap race lost, then no harm done
     (notice that (k,v2) could match in either diskMap or memMap,
     or both, or neither).
    3.1. Only expunge does an update of an existing diskMap entry for 
      concurrent methods.
    3.2. CachedBdbMap.get() must be able to return referent or otherwise
      effect a swap-in if get() is invoked during a swap out of a
      particular key-value entry.
 4. sync() to disk: synchronized sync() // fully locked, no races.
 

See the "find or create" style cache ServerCache.getServerFor(CandidateURI) for the major high-concurrency client of this class in terms of number of objects stored and performance impact.

refQueue

protected transient java.lang.ref.ReferenceQueue<V> refQueue

Deprecated.

diskMapSize

protected java.util.concurrent.atomic.AtomicInteger diskMapSize

Deprecated.

The number of objects in the diskMap StoredMap. (Package access for unit testing.)

referentField

protected static java.lang.reflect.Field referentField

Deprecated.

Reference to the Reference#referent Field.

canary

protected transient java.lang.ref.SoftReference<CachedBdbMap.LowMemoryCanary> canary

Deprecated.

Constructor Detail

CachedBdbMap

public CachedBdbMap(java.lang.String dbName)

Deprecated.

Constructor. You must call #initialize(Environment, Class, Class, StoredClassCatalog) to finish construction. Construction is two-stepped to support reconnecting a deserialized CachedBdbMap with its backing bdbje database.

Parameters:

dbName - Name of the backing db this instance should use.

CachedBdbMap

public CachedBdbMap(java.io.File dbDir,
                    java.lang.String dbName,
                    java.lang.Class<K> keyClass,
                    java.lang.Class<V> valueClass)
             throws com.sleepycat.je.DatabaseException

Deprecated.

A constructor for creating a new CachedBdbMap. Even though the put and get methods conforms to the Collections interface taking any object as key or value, you have to submit the class of the allowed key and value objects here and will get an exception if you try to put anything else in the map.

This constructor internally calls #initialize(Environment, Class, Class, StoredClassCatalog). Do not call initialize if you use this constructor.

Parameters:

dbDir - The directory where the database will be created.

dbName - The name of the database to back this map by.

keyClass - The class of the objects allowed as keys.

valueClass - The class of the objects allowed as values.

Throws:

com.sleepycat.je.DatabaseException - is thrown if the underlying BDB JE database throws an exception.

Method Detail

initialize

public void initialize(com.sleepycat.je.Environment env,
                       java.lang.Class<? super V> valueClass,
                       com.sleepycat.bind.serial.StoredClassCatalog classCatalog)
                throws com.sleepycat.je.DatabaseException

Deprecated.

Call this method when you have an instance when you used the default constructor or when you have a deserialized instance that you want to reconnect with an extant bdbje environment. Do not call this method if you used the CachedBdbMap(File, String, Class, Class) constructor.

Parameters:

env -

keyClass -

valueClass -

classCatalog -

Throws:

com.sleepycat.je.DatabaseException

initializeInstance

protected void initializeInstance()

Deprecated.

Do any instance setup. This method is used by constructors and when deserializing an instance.

initTransientStats

protected void initTransientStats()

Deprecated.

createDiskMap

protected com.sleepycat.collections.StoredSortedMap createDiskMap(com.sleepycat.je.Database database,
                                                                  com.sleepycat.bind.serial.StoredClassCatalog classCatalog,
                                                                  java.lang.Class keyClass,
                                                                  java.lang.Class valueClass)

Deprecated.

openDatabase

protected com.sleepycat.je.Database openDatabase(com.sleepycat.je.Environment environment,
                                                 java.lang.String dbName)
                                          throws com.sleepycat.je.DatabaseException

Deprecated.

Throws:

com.sleepycat.je.DatabaseException

close

public void close()

Deprecated.

Description copied from interface: ObjectIdentityCache

close/release any associated resources

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface ObjectIdentityCache<K,V>

finalize

protected void finalize()
                 throws java.lang.Throwable

Deprecated.

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable

keySet

public java.util.Set<K> keySet()

Deprecated.

The keySet of the diskMap is all relevant keys.

Specified by:

keySet in interface java.util.Map<K,V>

Specified by:

keySet in interface ObjectIdentityCache<K,V>

Overrides:

keySet in class java.util.AbstractMap<K,V>

See Also:

Map.keySet()

entrySet

public java.util.Set<java.util.Map.Entry<K,V>> entrySet()

Deprecated.

Specified by:

entrySet in interface java.util.Map<K,V>

Specified by:

entrySet in class java.util.AbstractMap<K,V>

getOrUse

public V getOrUse(K key,
                  Supplier<V> supplierOrNull)

Deprecated.

ObjectIdentityCache get-or-atomic-create method.

Specified by:

getOrUse in interface ObjectIdentityCache<K,V>

get

public V get(java.lang.Object object)

Deprecated.

Description copied from interface: ObjectIdentityCache

get the object under the given key/name

Specified by:

get in interface java.util.Map<K,V>

Specified by:

get in interface ObjectIdentityCache<K,V>

Overrides:

get in class java.util.AbstractMap<K,V>

put

public V put(K key,
             V value)

Deprecated.

Map.put() implementation.

Warning: This method violates the "only expose one value instance" design rule for this class. Multiple instances means that it is undetermined as to which instance will be saved to disk last. This method can be safely used when only one thread has access to this instance of CachedBdbMap.

If possible, use putIfAbsent(K, V) instead.

 Preconditions: (diskMap.containsKey(), memMap.containsKey()) are:
 (*,*) put() cannot assure "only expose one value instance", so
     all cases are the same: put to diskMap then memMap; clear any 
     pre-existing SoftEntry to prevent expunge of old value.
 
 PostConditions:
 (T,T) both memMap and diskMap will have entries for given key; if 
     null is returned, then this is the first time the key has an
     entry, otherwise the previous value in diskMap will not be 
     updated until expunge. 
 

Specified by:

put in interface java.util.Map<K,V>

Overrides:

put in class java.util.AbstractMap<K,V>

Parameters:

key -

value -

Returns:

previous value or null.

replace

public boolean replace(K key,
                       V oldValue,
                       V newValue)

Deprecated.

Replace entry for key only if currently mapped to given value. To maintain the illusion of an invisible cache, if the memMap has no mapping, the diskMap entry, if any, must be swapped-in to memMap in order to see if the given oldValue matches.

Possible disk io to swap in from diskMap.

Warning: This method violates the "only expose one value instance" design rule for this class. Multiple instances means that it is undetermined as to which instance will be saved to disk last. This method can be safely used when only one thread has access to this instance of CachedBdbMap.

 Preconditions: (diskMap.containsKey(), memMap.containsKey()) are:
 (F,F) nothing to replace. 
    PostCondition: (F,F) no replace.
 (T,F) value must be swapped-in, do replace(), 
    PostCondition: (T,T) if
    replace occurred or not (value might have been replaced), diskMap
    need not be updated because expunge will do that.
 (*,T) normal swapped-in condition (we can assume that the method which
    put a mapping to memMap took care of creating one for diskMap), 
    memMap value is most recent and expunge can take care of 
    updating diskMap, so we only do replace() on memMap; 
    PostCondition: (*,T).
 

Specified by:

replace in interface java.util.concurrent.ConcurrentMap<K,V>

Returns:

true if the replace was peformed (only if the previous mapping of key was to given oldValue).

replace

public V replace(K key,
                 V value)

Deprecated.

Replace entry for key only if currently mapped to some value. To maintain the illusion of an invisible cache, the diskMap value must be read to see if oldValue matches. The replace() operation is performed on memCache and then on diskCache, without synchronizing over the pair of operations.

Possible disk io to swap in from diskMap.

Warning: This method violates the "only expose one value instance" design rule for this class. Multiple instances means that it is undetermined as to which instance will be saved to disk last. This method can be safely used when only one thread has access to this instance of CachedBdbMap.

 Preconditions: (diskMap.containsKey(), memMap.containsKey()) are:
 (F,F) nothing to replace. PostCondition: (F,F) no change.
 (T,F) value must be swapped-in, do replace(), PostCondition: (T,T) if
    replace occurred or not (value might have been replaced), diskMap
    need not be updated because expunge will do that.
 (*,T) normal swapped-in condition (we can assume that the method which
    put a mapping to memMap took care of creating one for diskMap), 
    memMap value is most recent and expunge can take care of updating 
    diskMap, so we only do replace() on memMap.
 

Specified by:

replace in interface java.util.concurrent.ConcurrentMap<K,V>

Returns:

previous value if the replace was peformed on either memMap or diskMap, the previous value in memMap is returned if non-null, otherwise the previous from diskMap is returned, if no match in either map, then null is returned.

putIfAbsent

public V putIfAbsent(K key,
                     V value)

Deprecated.

A composite putIfAbsent() over memMap and diskMap. If the specified key is not already associated with any value, associate it with the given value. By modifying memMap and diskMap.

diskMap is not updated every time a value is mutated because this cached disk store attempts to be invisible to client code. However, This method will cause diskMap to be updated if there was no previous because this was the case for v1.14.2 and earlier. (It also means diskMap and memMap, both ConcurrentMaps, can be coherently maintained from the perspective of a one-instance-only style cache with diskMap updated at expunge by always modifying diskMap first here. The disk map size estimate is also reasonably correct if diskMap always has any mapping that is in memMap.)

Swap in/First insert: diskMap.putIfAbsent(), then memMap.putIfAbsent(), putIfAbsent() assures atomicity;

 Preconditions: (diskMap.containsKey(), memMap.containsKey(SoftEntry)) are:
 (F,F) initial starting conditions: is absent, put to diskMap then memMap.
 (F,T) transient remove(): await other thread to finish with memMap, 
     then proceed as (F,F) 
       OR (if remove not used) an unexpected data race occurred.
 (T,F) reloadable from disk or in process of inserting first time:
     not absent.
 (T,T) normal swapped in condition: not absent.
 
 PostConditions:
 (T,T) both memMap and diskMap will have entries for given key (if 
     null is returned, then the value given is the one mapped).
 

Specified by:

putIfAbsent in interface java.util.concurrent.ConcurrentMap<K,V>

Parameters:

key -

value - the value to which the given key should map, but only if there is existing mapping for key.

Returns:

clear

public void clear()

Deprecated.

Note that a call to this method CLOSEs the underlying bdbje. This instance is no longer of any use. It must be re-initialized. We close the db here because if this BigMap is being treated as a plain Map, this is only opportunity for cleanup.

Specified by:

clear in interface java.util.Map<K,V>

Overrides:

clear in class java.util.AbstractMap<K,V>

remove

public V remove(java.lang.Object key)

Deprecated.

Remove mapping for the given key. A mapping does not need to be swapped in to be removed. Matching entry is removed from memMap and diskMap.

 Preconditions: (diskMap.containsKey(), memMap.containsKey(SoftEntry)) are:
 (F,*) nothing to remove if diskMap has no entry.
 (T,F) remove from diskMap, await expunge in memMap, if in progress.
 (T,T) remove from diskMap, clear phantom to prevent expunge of memMap.
 
 PostConditions:
 (F,F) both memMap and diskMap will NOT have entries for given key (if 
     null is returned, then nothing was removed).
 

Specified by:

remove in interface java.util.Map<K,V>

Overrides:

remove in class java.util.AbstractMap<K,V>

Parameters:

key -

Returns:

old value or null if nothing removed.

remove

public boolean remove(java.lang.Object key,
                      java.lang.Object value)

Deprecated.

remove item matching both the key and value. Matching entry is removed from memMap and diskMap. A mapping does not need to be swapped in to be removed. If the entry matching key is swapped in, then value is used to find and remove the entry in memMap (the value is of type V), and diskMap.remove(K) is used to remove the persistent entry. Otherwise, a diskMap.remove(K,V) is performed.

 Preconditions: (diskMap.containsKey() AND value.equals(V), 
        memMap.containsKey(SoftEntry) AND value.equals(V) AND
        swapped-in) are:
 (F,F) nothing to do.
 (F,T) diskMap.remove(K,V) fails, but memMap.remove(K,V) still possible
 (T,F) remove from diskMap, await expunge in memMap, if in progress.
 (T,T) remove from diskMap, clear phantom to prevent expunge of memMap.
 
 PostConditions:
 (F,F) both memMap and diskMap will NOT have entries for given key (if 
     false is returned, then nothing was removed).
 

Specified by:

remove in interface java.util.concurrent.ConcurrentMap<K,V>

Parameters:

key - key used for matching mapping to remove.

value - value used for matching mapping to remove.

Returns:

true if entry removed.

containsKey

public boolean containsKey(java.lang.Object key)

Deprecated.

Specified by:

containsKey in interface java.util.Map<K,V>

Overrides:

containsKey in class java.util.AbstractMap<K,V>

containsValue

public boolean containsValue(java.lang.Object value)

Deprecated.

This method is not supported.

Specified by:

containsValue in interface java.util.Map<K,V>

Overrides:

containsValue in class java.util.AbstractMap<K,V>

Parameters:

value -

Returns:

size

public int size()

Deprecated.

Description copied from interface: ObjectIdentityCache

count of name-to-object contained

Specified by:

size in interface java.util.Map<K,V>

Specified by:

size in interface ObjectIdentityCache<K,V>

Overrides:

size in class java.util.AbstractMap<K,V>

getDatabaseName

protected java.lang.String getDatabaseName()

Deprecated.

sync

public void sync()

Deprecated.

Sync in-memory map entries to backing disk store. When done, the memory map will be cleared and all entries stored on disk.

Specified by:

sync in interface ObjectIdentityCache<K,V>