[R1soft/CDP/Idera] The conglomerate (2,849) requested does not exist

This error was showing in the R1soft/CDP/Idera (whatever you want to call it) server.log when trying to pull all of our Disk Safe data using the API:

The conglomerate (2,849) requested does not exist

It was causing the API call to fail and our script to stop functioning.  After some investigation I deduced that R1soft uses Apache’s Derby Database system for storing disk safe meta data.  The error basically means that the index for the metadata is corrupted.  Rather than going through each disk safe’s meta data and check its index there is an easier way to find out which disk safe or safes are having the issue.  You just need to restart the cdp server:

/etc/rc.d/init.d/cdp-server stop

/etc/rc.d/init.d/cdp-server start

And then tail the cdp server log:

tail -f /usr/sbin/r1soft/log/server.log

The server will take a while on startup to start the storage cache and output this:

2015-08-06 13:33:41,744 INFO   – Storage Cache Service starting

This will output typically some info on rolling back incomplete transactions but what you’re looking for is something like this:

2015-08-06 13:04:22,267 ERROR  – Failed to update disk safe cache entry for disk safe at ‘/backups/Core_Servers/f1ea4e4b-5ab2-4986-bfa8-f277b51860f9′

javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not prepare statement

at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)

at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)

at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)

at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:108)

at com.r1soft.backup.server.disksafe.metadata.entity.RecoveryPoint.deleteMembershipEntitiesNotKeptInMerge(RecoveryPoint.java:1329)

at com.r1soft.backup.server.disksafe.metadata.entity.DiskSafeMetaData.cleanupRecoveryPoint(DiskSafeMetaData.java:1700)

at com.r1soft.backup.server.disksafe.metadata.entity.DiskSafeMetaData.cleanupRunningRecoveryPoints(DiskSafeMetaData.java:658)

at com.r1soft.backup.server.disksafe.metadata.DiskSafeManager.openDiskSafe(DiskSafeManager.java:214)

at com.r1soft.backup.server.facade.DiskSafeFacade.getDiskSafeData(DiskSafeFacade.java:1318)

at com.r1soft.backup.server.facade.DiskSafeFacade.getDiskSafeData(DiskSafeFacade.java:1285)

at com.r1soft.backup.server.cache.DiskSafeCacheEntry.update(DiskSafeCacheEntry.java:106)

at com.r1soft.backup.server.cache.DiskSafeCache.initCache(DiskSafeCache.java:188)

at com.r1soft.backup.server.cache.DiskSafeCache.start(DiskSafeCache.java:116)

at com.r1soft.StorageCacheWrapper.start(StorageCacheWrapper.java:76)

at com.r1soft.Main.Rb(Main.java:308)

at com.r1soft.Main.run(Main.java:173)

Caused by: org.hibernate.exception.GenericJDBCException: could not prepare statement

at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)

at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:188)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareStatement(StatementPreparerImpl.java:91)

at org.hibernate.engine.query.spi.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:196)

at org.hibernate.internal.SessionImpl.executeNativeUpdate(SessionImpl.java:1313)

at org.hibernate.internal.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:401)

at org.hibernate.ejb.QueryImpl.internalExecuteUpdate(QueryImpl.java:196)

at org.hibernate.internal.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:401)

at org.hibernate.ejb.QueryImpl.internalExecuteUpdate(QueryImpl.java:196)

at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:99)

… 12 more

Caused by: java.sql.SQLException: The conglomerate (11,393) requested does not exist.

at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)

at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)

at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)

at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)

at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)

at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)

at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:162)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$1.doPrepare(StatementPreparerImpl.java:98)

at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

… 18 more

Caused by: java.sql.SQLException: The conglomerate (11,393) requested does not exist.

at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)

at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)

… 34 more

Caused by: ERROR XSAI2: The conglomerate (11,393) requested does not exist.

at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)

at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source)

at org.apache.derby.impl.store.access.CacheableConglomerate.setIdentity(Unknown Source)

at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)

at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source)

at org.apache.derby.impl.store.access.RAMTransaction.findConglomerate(Unknown Source)

at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source)

at org.apache.derby.impl.store.access.RAMTransaction.openStoreCost(Unknown Source)

at org.apache.derby.impl.sql.compile.CompilerContextImpl.getStoreCostController(Unknown Source)

at org.apache.derby.impl.sql.compile.FromBaseTable.getStoreCostController(Unknown Source)

at org.apache.derby.impl.sql.compile.FromBaseTable.estimateCost(Unknown Source)

at org.apache.derby.impl.sql.compile.OptimizerImpl.estimateTotalCost(Unknown Source)

at org.apache.derby.impl.sql.compile.OptimizerImpl.costBasedCostOptimizable(Unknown Source)

at org.apache.derby.impl.sql.compile.OptimizerImpl.costOptimizable(Unknown Source)

at org.apache.derby.impl.sql.compile.FromBaseTable.optimizeIt(Unknown Source)

at org.apache.derby.impl.sql.compile.ProjectRestrictNode.optimizeIt(Unknown Source)

at org.apache.derby.impl.sql.compile.OptimizerImpl.costPermutation(Unknown Source)

at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source)

at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source)

at org.apache.derby.impl.sql.compile.DMLModStatementNode.optimizeStatement(Unknown Source)

at org.apache.derby.impl.sql.compile.DeleteNode.optimizeStatement(Unknown Source)

at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)

at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)

at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)

… 28 more

The very first line will give you the UUID of the disk safe with the corrupted meta data.  When we went to look at it in the web interface it was very obvious that it was corrupted.  See the image below and how the selected one lists no info.

Idera_Server_Backup_Manager_SE

Simply delete it and recreate it and you’re all set.