Category: Unity Express and Connection



* Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab

* Hint:Please double-click the snapshots to zoom into them. 

This is one of the most intriguing cases I have every dealt with:   This client has Cisco Unity connection 8.6x integrated with a Microsoft Forest containing  over ten Microsoft server 2003 and 2008 domains controllers spread out across multiple sites in Europe and the Americas.

The client complained that all of a sudden, when new users were created in Active directory, they would no longer show up in Cisco unity connection . I asked the client what had  changed prior to the problem and I was informed that nobody had touched the Cisco servers except for  the fact that their Group policies had been recently been re-written within active-directory.

I logged unto Cisco unity connection and tried do an import of users and nothing happened; no error messages were displayed on the webpage and no  newly created users from Active-Directory were imported.

I made up my mind that I would have to  pull out the ‘ Big-Guns’:

Stage one 

1)    I Installed Wireshark on a  PC and started capturing packets

Stage Two

On the same  PC, I also installed Softerra LDAP Browser. softerra LDAP Browser is a tool that you simple put the username, password and  IP-address of your LDAP server into and it will show you everything in your Domain. This was going to be the way that I would use to simply test the username and password that Cisco unity connection was using.   If you need this tool, here is the link to it: http://www.ldapbrowser.com/download.htm

Here is snapshot  of the tool in action:

LDAP-Browser

Stage Three

After I entered the username,password and IP-address of the LDAP server, I started seeing errors in the Wireshark. Here is a screen-shot of the errors below:

LDAP bind failure

The username and password that I was using to authenticated was the same one that the Cisco unit Connection server was using so from the wireshark capture, it was clear that the reason the unity connection server was unable to import users from Active Directory was because it did not even make it pass authentication let-alone pulling user database.

If you look at the error message in the capture above, it say: ‘ In order to perform this operation, a successful bind must be completed on the connection’  

The term ‘bind’ means authentication. So in ordinary English, the error message that can be seen in the LDAP message coming back from the LDAP server  is basically saying: in order to search the LDAP user database, you must first authenticate successfully.

I double checked with  the ‘DirSync’ service traces on unity connection and I found exactly the same error message.

Stage Four

With this evidence, I told the client to speak to his Server team; who in tern, contacted Microsoft Directly. Shortly after the Microsoft team arrived on the scene, they informed me that they did not share my view that the problem was on the Microsoft server side.  They stated that since other  users and servers were authenticating against LDAP without any problems, the LDAP server was fine.

Stage Five:

I knew I was talking to skilled and experienced Engineers so I was not quick to doubt them. I pulled out my wireshark again and investigated how the other non-cisco servers were managing to authenticate without any problems. what I found was that Cisco Unity Connection was using a method of authentication called ‘ Simple Bind‘ while the other servers were not.  Simple bind authentication method is when  usernames and passwords are sent in clear text: I will share how I got to know this later in this post.

I went back to Microsoft and informed them that they  needed to permit simple bind for the Cisco Unity Connection server because that is how Unity connection operates. And that if they wanted security, they would have to use ‘ transport-Layer-security (TLS)’ because that is what Unity Connection supports.

Stage Six: 

In order to test my theory, Microsoft then decided to enable Anonymous Bind for test purposes only. Anonymous bind is when a client can authenticate to the server without username or password.

 

Stage Seven

After we made the change above, the Cisco unity Connection still could not import users from Microsoft Active directory. Again, the Microsoft Team told me that they did not share my view . However, I reminded them that  I  asked for ‘simple bind’ to be allowed on the network not ‘ anonymous bind’ so they continued to work with me.

I was really starting to look really  bad.  I pulled out wireshark again and captured the packets being exchanged between the Cisco Unity Connection server and Microsoft LDAP Server and noticed that something had changed since anonymous bind was allowed;  this is what I found:

LDAP_no such object.

Here is the interpretation of the logs above:

1)  In the very first line, we see that Cisco Unity Connection server at Ip-address 172.16.6.33 sends an LDAP simple bind request to the LDAP server at IP-address  172.16.6.5. We know that we are using simple bind authentication because can see the word ‘ simple‘ at the very end of the line.

2)  Looking at the ‘ source’ and ‘destination’  fields In the second line of the capture, we can see that the LDAP server sends a message back to the unity connection server stating that authentication had successfully completed.

3)  In the third line, we see  the Unity connection server trying to find all the users in the database by sending a search request to the LDAP sever and in the fourth line we see that the server responds with a result of zero; meaning that no result was provided to the Cisco unity connection server.

4)  Going further down the line, we see that the unity connection server continues trying until the 20th line when the LdAP Server responds back with NameErr  0000208D, Problem 2001 (No_object)

Stage Eight

I decided to collect traces  from Unity Connection  so I turned on detailed traces, performed a full LDAP system sync on unity connection and collected Cisco Dir-sync service traces from the unity connection server. Here is what I  found:

DAPSync(cca4b920-9361-9535-8d80-ece481332007)[getInvocationId] caught exception … [LDAP: error code 32 – 0000208D: NameErr: DSID-03151F00, problem 2001 (NO_OBJECT)

 According to the following Microsoft website, LDAP error code 32 means:  ‘The user has insufficient access rights: http://support.microsoft.com/kb/218185

Stage nine

I Informed  the Server Admins and  Microsoft team about the new error messages I was getting from their Microsoft servers.  I asked the team to explain why their server was not happy with my request since the account I was using to authenticate was an admin user. In response, they informed me that the server was treating my Simple bind Request as an Anonymous bind request and since Anonymous bind request were not permitted to search the database, my request were being dropped.  To remedy this, the following steps were applied to the Domain Controller.

  1. Open regedit.exe on a domain controller
  2. Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders
  3. In the right window pane double-click on SecurityProviders
  4. Add pwdssp.dll to the end of the Value data field which should be separated by a comma after the last entry.  For example:

msapsspc.dll, schannel.dll, digest.dll, msnsspc.dll, pwdssp.dll

5.  Reboot the DC: This was done on all the domain controllers on Windows 2008 and above and a reboot was required

Stage Ten

After the changes above were applied, everything started working fine  lol 🙂 . . . and all along, everyone was blaming me but thanks to wireshark, I was able to prove to the remote Microsoft Server team that the problem was from them.  Long live Wireshark. 🙂 and Mr Remi A @silvercleric for his contribution to this case.

Advertisements

 * Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab

 * Hint: Please click on the pictures in order to maximize them

This was a case that turned out to be another unexpected ticket which had a few gatchas up it’s sleeves.  The issue started off with a client side engineer stating that he could not log into one of the nodes of his Unity Connection cluster.  As soon as I got this information, I Jumped on the cluster in order to see what was happening.

I tried logging into the Web graphical user interface but I got an error message instead. The error message said; ‘database communication error’. I logged unto the subscriber server and that was fine.  I then decided to log into the Publisher server via the command line interface.

As soon as I logged on, I saw these errors:

Command Line Interface is starting up, please wait …
java.io.FileNotFoundException: /var/log/active/platform/log/cli.bin (Read-only file system)
at java.io.RandomAccessFile.open(Native Method)

log4j:ERROR No output stream or file set for the appender named [CLI_LOG].
Welcome to the Platform Command Line Interface

The /common file system is mounted read only.
Please use Recovery Disk to check the file system using fsck
.

At this point, it was clear that the system had gone into read-only-mode and would have to be recovered. The team agreed to recover this server whilst planning a full rebuild.

After the server was recovered, two days later it crashed again so a rebuild was now the only reasonable thing to do. The responsibility fell on me so I rushed to create a new VMware Virtual Unity Connection server. In approaching this case, I created a plan that ensured that I was following Cisco’s best practice recommendations and that I would be able to kick-start the new server and everything would work perfectly.  My plan was to make sure that the new server was as similar to the old one as possible so I made sure the following variables were the same:

1)      Server Hard-drive size

2)      Server memory size

3)      Server NTP Server

4)      Server IP –address

5)      Server Default-Gateway IP

6)      Cluster  security password

7)      Same server version as the old server

8)      Server host-name

Whilst it’s not necessary to use an OVA Template, I decided to use the template shown in the screenshot below as Unity ova template formats the drive in 64kB Blocks which results in ‘ improved storage input/output operation.

one

It is very crucial for the hostname to be the same as the failed server otherwise; the subscriber will complain that the new publisher’s host-file does not match the host-file that it has in its database. So what was needed was to basically trick the subscriber into believing that its connected to the original Publisher server when we connect them up.

So I loaded the Unity Connection installer and started building. As can been seen in the screenshot below, after a while, I got to the part where I had to enter the cluster certificate information.

two

As I did not have this information readily at hand and I was paranoid about build a near exact replica, I logged unto the subscriber unity connection server and then navigated to

i)                    OS admin page

ii)                   Security

iii)                 Certificate management

iv)                 And I selected the tomcat trust certificate

From the screen-shot below, did you noticed that all the information that was required of me in the screenshot above was actually  listed screenshot  below?

three

After filling in the required fields, I selected ‘ next’ on install process.

I was then asked by the install script if this was the first server in the cluster: I selected yes.

After while, the installation process seemed to have frozen or hung.  After a quick rummaging of the internet, I discovered that the server was actually installing in the background: The server appeared to have frozen because it was experiencing Bug  CSCtj33840 so I ignored it and after a while the installation completed.

After the server had rebooted, I followed Cisco’s upgrade guide and did the following.


Step 1 Sign in to Cisco Unity Connection Administration on the publisher server.

Step 2 In Cisco Unity Connection Administration, expand System Settings, then select Cluster.

Step 3 On the Find and List Servers page, select Add New.

Step 4 On the New Server Configuration page, in the Hostname/IP Address field, enter the hostname or IP address of the subscriber server.

Step 5 In the Description field, enter a description for the subscriber server.

Step 6 Select Save.

Step 7 Sign out of Cisco Unity Connection Administration.


To Connect the Subscriber Server to the New Connection Cluster, and Replicate Data and Messages to the Publisher Server


Step 1 Sign in to the command-line interface (CLI) for the subscriber server.

Step 2 Run the CLI command utils cuc cluster renegotiate.

After issuing this command, I got the output below:

 

This command may take require several hours to complete. The duration 

depends on the network bandwidth between servers and on the

total size of the data on this subscriber server.

 

Are you sure you want to continue? (y/n) : y

 

13/03/03 08:31:47  Disabling data replication…

13/03/03 08:31:57  Renegotiating ssh trusts…

13/03/03 08:31:59  Syncronizing platform and LDAP database…

13/03/03 08:35:07  Creating any missing messaging databases on the publisher…

13/03/03 08:35:10  Adding subscriber node to publisher…

13/03/03 08:35:12  Synchronizing Unity Connection databases…

13/03/03 08:41:05  Synchronizing file systems…

13/03/03 08:41:06  Synchronizing message files for mail store UnityMbxDb1…

13/03/03 08:41:07  Copying cluster DSCP configuration to publisher node…

13/03/03 08:41:08  Rebooting publisher node v8-unity-conn-192-168-0-56…

13/03/03 08:41:10  Cluster renegotiation completed.

 

Process completed. 

 

After the completion, I checked the cluster status by issuing the command ‘ show cuc cluster status’ and everything was fine.

The next phase was to re-host the license files from the old server to the new server but in order to do this; I needed the value of the license-mac of the new server. The information below proved crucial.

Host Name    : publisher-cuc

Date         : Sat Mar 2, 2013 09:34:39

Time Zone    : Greenwich Mean Time (Europe/London)

Locale       : en_US.UTF-8

Product Ver  : 7.1.5.30000-1

Platform Ver : 2.0.0.1-1

License MAC  : 0ce90be10d98

As can be seen above, I logged into the command line interface of the new publisher and issued the command ‘show status’. I then copied and sent the new ‘License-Mac’ to Cisco who then sent me the new license file which I then installed.

The server has been happy ever since. . .


If you are visiting for the first time then I would like to say welcome to Voice Initiate. It is a place where experiences are shared and discussed.  As we all know, working in technology is all about being introduced to new things on an almost daily basis. For me, it’s like being in a perpetual state of initiation.

If you are reading this blog then I’m sure you can relate when I say that I came into the IP telephony world not knowing that I would have to learn programming too, but working on Contact Centers dictates  that  you must- so retreat was never an option. . . it’s an endless battle but we love it anyway.

This blog is an account of the joys and challenges faced by a Unified Communications NOC Engineer. This is a  place where both failure and success is discussed openly.

Advice and contributions are always welcomed.  My hope is that this blogs brings a sense of thrill to fellow bloggers and initiates alike and a sense of nostalgia to those that have seen the joys and challenges that we now speak of .

Cheers!

maxwell.osagie@yahoo.com

%d bloggers like this: