This error was showing in the R1soft/CDP/Idera (whatever you want to call it) server.log when trying to pull all of our Disk Safe data using the API:
The conglomerate (2,849) requested does not exist
It was causing the API call to fail and our script to stop functioning. After some investigation I deduced that R1soft uses Apache’s Derby Database system for storing disk safe meta data. The error basically means that the index for the metadata is corrupted. Rather than going through each disk safe’s meta data and check its index there is an easier way to find out which disk safe or safes are having the issue. You just need to restart the cdp server:
/etc/rc.d/init.d/cdp-server stop
/etc/rc.d/init.d/cdp-server start
And then tail the cdp server log:
tail -f /usr/sbin/r1soft/log/server.log
The server will take a while on startup to start the storage cache and output this:
2015-08-06 13:33:41,744 INFO – Storage Cache Service starting
This will output typically some info on rolling back incomplete transactions but what you’re looking for is something like this:
2015-08-06 13:04:22,267 ERROR – Failed to update disk safe cache entry for disk safe at ‘/backups/Core_Servers/f1ea4e4b-5ab2-4986-bfa8-f277b51860f9’
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397)
at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:108)
at com.r1soft.backup.server.disksafe.metadata.entity.RecoveryPoint.deleteMembershipEntitiesNotKeptInMerge(RecoveryPoint.java:1329)
at com.r1soft.backup.server.disksafe.metadata.entity.DiskSafeMetaData.cleanupRecoveryPoint(DiskSafeMetaData.java:1700)
at com.r1soft.backup.server.disksafe.metadata.entity.DiskSafeMetaData.cleanupRunningRecoveryPoints(DiskSafeMetaData.java:658)
at com.r1soft.backup.server.disksafe.metadata.DiskSafeManager.openDiskSafe(DiskSafeManager.java:214)
at com.r1soft.backup.server.facade.DiskSafeFacade.getDiskSafeData(DiskSafeFacade.java:1318)
at com.r1soft.backup.server.facade.DiskSafeFacade.getDiskSafeData(DiskSafeFacade.java:1285)
at com.r1soft.backup.server.cache.DiskSafeCacheEntry.update(DiskSafeCacheEntry.java:106)
at com.r1soft.backup.server.cache.DiskSafeCache.initCache(DiskSafeCache.java:188)
at com.r1soft.backup.server.cache.DiskSafeCache.start(DiskSafeCache.java:116)
at com.r1soft.StorageCacheWrapper.start(StorageCacheWrapper.java:76)
at com.r1soft.Main.Rb(Main.java:308)
at com.r1soft.Main.run(Main.java:173)
Caused by: org.hibernate.exception.GenericJDBCException: could not prepare statement
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:188)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareStatement(StatementPreparerImpl.java:91)
at org.hibernate.engine.query.spi.NativeSQLQueryPlan.performExecuteUpdate(NativeSQLQueryPlan.java:196)
at org.hibernate.internal.SessionImpl.executeNativeUpdate(SessionImpl.java:1313)
at org.hibernate.internal.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:401)
at org.hibernate.ejb.QueryImpl.internalExecuteUpdate(QueryImpl.java:196)
at org.hibernate.internal.SQLQueryImpl.executeUpdate(SQLQueryImpl.java:401)
at org.hibernate.ejb.QueryImpl.internalExecuteUpdate(QueryImpl.java:196)
at org.hibernate.ejb.AbstractQueryImpl.executeUpdate(AbstractQueryImpl.java:99)
… 12 more
Caused by: java.sql.SQLException: The conglomerate (11,393) requested does not exist.
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
at com.mchange.v2.c3p0.impl.NewProxyConnection.prepareStatement(NewProxyConnection.java:162)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$1.doPrepare(StatementPreparerImpl.java:98)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)
… 18 more
Caused by: java.sql.SQLException: The conglomerate (11,393) requested does not exist.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
… 34 more
Caused by: ERROR XSAI2: The conglomerate (11,393) requested does not exist.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.store.access.btree.index.B2IFactory.readConglomerate(Unknown Source)
at org.apache.derby.impl.store.access.CacheableConglomerate.setIdentity(Unknown Source)
at org.apache.derby.impl.services.cache.ConcurrentCache.find(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source)
at org.apache.derby.impl.store.access.RAMTransaction.findConglomerate(Unknown Source)
at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source)
at org.apache.derby.impl.store.access.RAMTransaction.openStoreCost(Unknown Source)
at org.apache.derby.impl.sql.compile.CompilerContextImpl.getStoreCostController(Unknown Source)
at org.apache.derby.impl.sql.compile.FromBaseTable.getStoreCostController(Unknown Source)
at org.apache.derby.impl.sql.compile.FromBaseTable.estimateCost(Unknown Source)
at org.apache.derby.impl.sql.compile.OptimizerImpl.estimateTotalCost(Unknown Source)
at org.apache.derby.impl.sql.compile.OptimizerImpl.costBasedCostOptimizable(Unknown Source)
at org.apache.derby.impl.sql.compile.OptimizerImpl.costOptimizable(Unknown Source)
at org.apache.derby.impl.sql.compile.FromBaseTable.optimizeIt(Unknown Source)
at org.apache.derby.impl.sql.compile.ProjectRestrictNode.optimizeIt(Unknown Source)
at org.apache.derby.impl.sql.compile.OptimizerImpl.costPermutation(Unknown Source)
at org.apache.derby.impl.sql.compile.SelectNode.optimize(Unknown Source)
at org.apache.derby.impl.sql.compile.DMLStatementNode.optimizeStatement(Unknown Source)
at org.apache.derby.impl.sql.compile.DMLModStatementNode.optimizeStatement(Unknown Source)
at org.apache.derby.impl.sql.compile.DeleteNode.optimizeStatement(Unknown Source)
at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown Source)
at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown Source)
… 28 more
The very first line will give you the UUID of the disk safe with the corrupted meta data. When we went to look at it in the web interface it was very obvious that it was corrupted. See the image below and how the selected one lists no info.
Simply delete it and recreate it and you’re all set.