No Server-side object found exception

[jtaeubrich]

Hi,

we're running RCF-2.0.0.2685 on Linux and in a part of our application we get the following exception:

• Dynamic exception type: N3RCF15RemoteExceptionE
  std::exception::what: [Remote][25: Server-side object not found. Service: %1. Interface: %2. Function id: 2.][0: No sub system.][0: ][What: [25: Server-side object not found. Service: %1. Interface: %2. Function id: 2.][0: No sub system.][0: ][What: ][Context: RcfServer.cpp:1129]][Context: RcfServer.cpp:1129]

From the log of the exception and the single place where the exception is thrown it seems that the service and subinterface is the empty string. Unfortunately I can't figure out why this could happen.
I tried to reproduce this behavior in a local unit test, but failed. On our target it appears while having quite some RCF traffic, but not all remote calls fail. It only seem to affect some of them.

I would appreciate any help to describe the scenario when this could happen.

Regards,
Jan

[jarl]

Hi Jan,

This is typically due to an automatic reconnect taking place. When a reconnect happens, session objects that were associated with the previous connections are deleted, and will no longer be present.

On your RcfClient<>, you can disable automatic reconnects:

Code: Select all

RcfClient<...> client;
client.getClientStub().setAutoReconnect(false);

, and that way you'll get an exception when the underlying TCP connection goes down, and at that point you can explicitly set up your connection again.

[jtaeubrich]

Hi Jarl,

thanks for your answer. Unfortunately this doesn't seem to fix it.
I see the exception thrown in the very first remote call performed via an RCF client. This call registers an RCF client with a name stored in a session object. There is no indication that the creation of the session object failed.

Do you have an idea to reproduce this locally?

Regards,
Jan

[jtaeubrich]

Hi Jarl,

I put a debug version of RCF on our target and tried to find out why the stub entry pointer is null, which seems to be the only reason for this exception.
The relevant code in the MethodInvocation looks like this...

Code: Select all

if (!targetName.empty())
{
	ReadLock readLock(rcfServer.mStubMapMutex);
	const std::string & servantName = getService();
	RcfServer::StubMap::iterator iter = rcfServer.mStubMap.find(servantName);
	if (iter != rcfServer.mStubMap.end())
	{
		stubEntryPtr = (*iter).second;
	}
}
else
{
	stubEntryPtr = pRcfSession->getDefaultStubEntryPtr();
}

return stubEntryPtr;

I set breakpoints on

Code: Select all

iter != rcfServer.mStubMap.end()

and in the outer else branch on

Code: Select all

stubEntryPtr = pRcfSession->getDefaultStubEntryPtr()

.
Running with this setup it looks like the targetName is empty somehow and hence the default stub entry is taken, which seems to be null.

If I locally force a test case into this else branch I get the same exception with the difference that the printout looks like this

RCF::RemoteException'
what(): [Remote][25: Server-side object not found. Service: I_AlertClient. Interface: I_AlertClient. Function id: 2.][0: No sub system.][0: ][What: [25: Server-side object not found. Service: I_AlertClient. Interface: I_AlertClient. Function id: 2.][0: No sub system.][0: ][What: ][Context: RcfServer.cpp:1129]][Context: RcfServer.cpp:1129]

Compared to the exception on our target system this supports the idea that the request's service and sub interface is empty.

EDIT:
I see the missing interface and sub interface already on a wireshark trace like if the mistake happens in the serialization.
Regards,
Jan

[jarl]

Can you simplify your client side code as far as possible, and send it to us?

[jtaeubrich]

Hi,

I was able to track down the problem to code independent from ours.
The following piece of code terminates with the known exception.

Code: Select all

#include <string>
#include <iostream>

#include <RCF/RCF.hpp>
#include <boost/container/flat_map.hpp>
#include <boost/lexical_cast.hpp>

RCF_BEGIN(I_AlertClient, "I_AlertClient")
   // send AlertStatusNotification messages
   RCF_METHOD_V0(void, sendAlertStatusNotification)
   // register alert client
   RCF_METHOD_V1(void, registerAlertClient, std::string const &)
RCF_END(I_AlertClient)

class SimpleAlertClientReceiver
{
public:
   void sendAlertStatusNotification() { std::cout << "Status!" << std::endl; }
   void registerAlertClient(std::string const &name) { std::cout << "Register: " << name << std::endl; }
};

boost::container::flat_map<std::string, RcfClient<I_AlertClient> > m_clients {};

RcfClient<I_AlertClient> &getClient(std::string const &alertClient)
{
   std::cout << "GetClient for " << alertClient << std::endl;
   if (m_clients.find(alertClient) == m_clients.end())
   {
      RcfClient<I_AlertClient> client(RCF::TcpEndpoint{ "localhost", 10052 });
      m_clients[alertClient] = client;
      m_clients[alertClient].getClientStub().connect();
      m_clients[alertClient].registerAlertClient(alertClient);
   }
   return m_clients[alertClient];
}

int main()
{
   RCF::RcfInitDeinit deinit {};
   RCF::RcfServer server{ RCF::TcpEndpoint{ "localhost", 10052 }};

   SimpleAlertClientReceiver sacr {};
   server.bind<I_AlertClient>(sacr);
   server.start();

   for (unsigned int i = 0; i < 20; ++i)
   {
      getClient("Client" + boost::lexical_cast<std::string>(i)).sendAlertStatusNotification();
      getClient("Client" + boost::lexical_cast<std::string>(i)).sendAlertStatusNotification();
   }

   server.stop();

   return 0;
}

The problem disappears if the getClient() function returns a shared pointer to an client instead of a reference. That saves some assignment operations.
I tried to debug the RcfClient's interface name, by replacing the macros with the preprocessed code, but the modifications we saw didn't seem reasonable for me.
But the problem seems to be related to the assignment in the getClient function. On my machine exception occurs on for i=18, all the time.

EDIT: I further found out that the boost::container::flat_map plays a key role. During the assignment

m_clients[alertClient] = client;

the mInterfaceName of the
default constructed lhs is overwritten with the empty string. Using a std::map doesn't reveal the issue. Since the performance of the flat_map is crucial to us I also tried to fix the assignment operator
from

const std::string &targetName = mInterfaceName;

to

const std::string &targetName = rhs.mInterfaceName;

. This also solved the problem.
Finally I'll stay with the shared pointers for now to avoid assignment at all. But maybe the assignment operator is worth to have a look at.

Regards,
Jan

[jarl]

I'm pretty sure the problem is that your map has this type:

Code: Select all

boost::container::flat_map<std::string, RcfClient<I_AlertClient> > m_clients;

I don't know how flat_map is implemented, but defining m_clients like this means that the RcfClient<> objects inside may be copied, when the container is modified.

When a RcfClient<> object is copied, the network connection itself is not copied, and the new copy has to establish its own connection, which is most likely not what you want.

To avoid this, you can use shared_ptr in the flat_map:

Code: Select all

typedef boost::shared_ptr< RcfClient<I_AlertClient> > 		AlertClientPtr;
boost::container::flat_map<std::string, AlertClientPtr > 	m_clients;

Also, if you have multiple threads in play, make sure that only one thread at a time is accessing any given RcfClient<>.

[jtaeubrich]

Hi Jarl,

thanks for the reply. It absolutely matches our observation and we don't want new connections on copy. Therefore, using shared pointers is the right solution for us and probably it could be even a general suggestion to use a shared pointer if session objects store connection related data.
But still I think the copy construction is not working correctly at least with boost flat_maps. The copied object is not able to establish its own connection as it has an empty interface name.
Do you agree to this and would consider fixing the copy constructor as I have sketched above (or somehow different) ?

Regards,
Jan

[jarl]

I was just looking now at the copy constructor you mentioned... You're right, that code is definitely incorrect - I've fixed it now.