Tag Archive: cisco



Hint: Double-click the pictures to zoom in

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

I thought I’d write about a recent issue I observed after there was a sudden power outage to a CVP server which is part of a UCCE 9 estate .

It was noticed that when calls were placed into the UCCE 9.x  contact centre, the calls would never connect; all that would be heard was a fast busy tone before the call was dropped.  So, I  checked the CUBE, PG, ROGGER and CVP and found that all the devices were up  and showing a status of full-service.  However, when I placed the ICM script into debug or monitor mode, I noticed that the calls were failing at the ” run external script” node of the ICM script.

This confirmed two things;

  1. That the problem was on the VRU leg since it was the switch leg that took the calls up to that point in the ICM script.
  2. That  I would probably find the root cause by looking at the VXML gateway and CVP logs.

One of the very important messages that the VXLM gateway sends to the CVP is the Ping message.  In the first screenshot below, notice the highlighted error message as the VXML gateway tried to interact with the CVP server via the http ping/call message? Also, in the second screenshot, notice how the VXML gateway thinks that the CVP server is out of service? This is because the CVP server was not responding even though the CVP server was showing a status of  full-service (in service)on the Operations Console.

At  this stage, I knew that even though the route to  finding the location of the problem was technical, the solution was going to be a very non-technical one indeed.   I guess at this stage you’ve guessed that the solution was 🙂 ? if you haven’t, the solution was a  simple reboot of the CVP server.  And this action  makes sense too because remember I started of by saying that the server had just be ungracefully shut-down due to a  power outage? So this highlights one  very important fix for UCCE: if you know the problem  you are troubleshooting was caused by a power outage, try rebooting the affected server as a first step.

Cheers,

Hope this helps someone out there.

bad fetch

call server cannot be reached

 

 

 

 

Advertisements

[]  The rise of Video conferencing as a preferred means of collaboration advocates for continuous  conversations on the issue of security. This entry seeks to contribute to that discussion.

Hope you find this useful.

 

 


Hello and welcome to this entry on Jabber guest. ‘Cisco Jabber Guest is a [Web-Based ] consumer-to-business (C2B) solution that extends the reach of a company’s internal enterprise telephony to people outside of the corporate firewall’ via the aid of a link that is posted or published on the company website. Thus, without any specialised hardware, jabber guest turns a regular webpage into a video/collaborative end-point at the moment that a consumer clicks on the web-based link/hyper-link.   A customer or client is now able to establish high-definition video communication with someone stationed within the internal network or even at a remote location by using a simple browser!

Jabber guest links/hyperlinks can also be embedded into documents and custom apps.

Please view the link below for a quick yet very detailed introduction.

http://youtu.be/n-USuvpNC6c

The video Demonstration  below is part one of two parts.

1) This part (part one) will focus on deploying a  Jabber guest cluster.
2) And part two will focus on integrating the jabber guest cluster with an Expressway cluster
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Today’s outline.
:::::::::::::::::::::::::::

1) Today I will be deploying Cisco Jabber guest in a cluster of three servers.

3) Then I will configure sip trunks between jabber guest and CUCM

4) After configuring the Jabber guest links on the Jabber guest server, I will then advertise/publish the links to a website.

5) I will then browse to the webpage and click on the link (that was published in step 4) in order to establish a video call between the web-based caller (i.e me) and the internal called device (jabber for windows soft-phone)

The result will be a call between a browser and a jabber for windows client. Enjoy!

 

 

 

 

Thanks for visiting

 

Regards!


Hint: Somewhere around 19 min into the recording I mentioned ‘server signing authority’ instead of ‘Certificate signing authority’.  Please process the information accordingly.

The promise of Mobile and remote access technology is that , using the Expressway-C and E servers, external devices are able to register to the corporate network  and  gain access to services  which are located within the corporate LAN without need for a VPN.  Internal services like voice-mail, directory, audio- video calls, on-premises instant message and presence information become transparently and seamlessly available to a mobile and remote devices as they move in and out of the network with no extra user education or involvement.

In the first half this video demonstration, I will:

i) Walk through what we already have configured on the corporate LAN.

In the second half, :

i) I will first of all install the Expressway-C and establish communication between Expressway-C with the internal servers like CUCM and IM&P.

ii) Then the expressway-C  is  connected to the Expressway-E via a traversal zone. The Expressway-E sits in the DMZ. The traversal zone is the link between the internal and external network.

iii) Finally, a jabber client is registered to the network via the Expressway-E. The jabber client is able to access internal services through the aid of the traversal zone that exist between the Expressway-E server that is  on the outside or DMZ of the network  and the Expressway-C server which is inside the network. .

Enjoy

 


Hint: As long as you are running a TC software that is from version 6.1 and above, you definitely have to make this change.

Hint: This work-around will not solve all call merge issues as a lot of call merge issues are ‘timer’ related. This solution is for Bug :  CSCug95623

Hint: I have also seen this error in the logs taken from endpoints that have their Ethernet settings set to auto instead of full.

Today I will be sharing a case I recently worked on  that involved several SX20 codecs, Cisco Video Communication servers and one Multipoint Control Unite (MCU). The problem was that  calls were failing to merge . As Network Professionals we know that a number of things can cause video conference calls to fail so what I’m going to do today is to first of all talk about the symptoms of this case  before talking about the solution. After talking about the solution, I will discuss how to identify this problem when looking at the logs.

Symptoms:

::::::::::::::::::::::::::::::

1)     A caller calls another video endpoint and then calls a third endpoint/person while the other caller is placed on hold.

2)     The conference initiator presses the merge button in an attempt to merge all three participants into one call. Normally, what happens in the background when a conference initiator presses the merge button is that all participants get redirected to the Conference Bridge (simplified explanation) or Multipoint Control Unit (MCU).

3)     However, on this occasion, the initiator simply saw a message saying ‘merge failed’ and the person that  was placed on hold before; remained on hold.  Please take note of the symptom that I have just explained because it is really crucial to knowing where the fault is.  Let me explain further: A few weeks ago I worked on another case where endpoints could not initiate a video conference (MCU conference). On that occasion, when the conference initiator pressed the merge button and everyone got redirected to the MCU; a ‘merge failed’ message was displayed and everyone got disconnected and no one remained on hold. This  failure was due to an incompatible sip timer issue between the endpoints and the MCU. But in this case that I am talking about in this blog, I noticed that the call remained on hold. Does this not make you think that the call was never redirected to the MCU because the first caller never left its ‘hold’ state that it was placed in before the call merge attempt? If you did not get that question, after reading the rest of this blog, please come back to this question and it will make sense. It is very important and in everyone’s benefit that anyone interested in this post is able to answer this question.

Solution:

:::::::::::::::::::::::::

Let me now provide the solution. The solution is to change the Sip profile of the Video endpoint from ‘ Cisco’ to ‘ Standard’.

1)     Log into the Video endpoint (eg SX20),  click the ‘ configuration’  menu

2)     Select ‘ system configuration’

3)     Select ‘ SIP’

4)     Select ‘standard’ as the ‘ Type’ instead of ‘ Cisco’ Standard Log analysis and detailed explanation:

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

As I mentioned before, when a conference initiator presses the merge button, a request is sent to the Video communication server with the multiway address as the destination. The video communication server understands that this is  conference request so   the video communication server checks the ‘conference factory’ configuration and responds back by sending back an address for which all conference participants  should  re-converge at. This address will be an address on the Multipoint Control Unit (MCU). The summary of the logs below shows  that  the  conference-initiating-endpoint  sends as  an incorrect multiway address to the VCS  even though the multiway address is configured correctly on the endpoint. So in response, the VCS basically comes back and tells the endpoint that it does not know that address so the call (merge) fails. These logs were taken from an SX20

Here is what the problem looks like in the logs

Jan  8 13:56:39.483 a8 appl[1592]: 1749764.49 MainEvents I: GenericTransferEvent() ‘MultiwayStarted

Jan  8 13:56:39.509 a8 appl[1592]: 1749764.52 SipPacket I: PacketDump: Proto: SIP, Direction: Outgoing, Name: REFER, CSeq: 101 REFER, RemoteAddress: 10.86.24.90:5061, CallId: 55c9039b4884ec8f0c834997b05cfa10,Time: 1749764517 Jan  8 13:56:39.520 a8 appl[1592]: 1749764.53 SipPacket   Call-ID: 55c9039b4884ec8f0c834997b05cfa10

Jan  8 13:56:39.520 a8 appl[1592]: 1749764.53 SipPacket   CSeq: 101 REFER Jan  8 13:56:39.525 a8 appl[1592]: 1749764.53 SipPacket   Content-Id: <3ac0bec2e9862d4b@127.0.0.1>

Jan  8 13:56:39.527 a8 appl[1592]: 1749764.54 SipPacket   Contact: <sip:812000@voiceinitiate.com;gr=urn:uuid:00000000-0000-0000-0000-d867d97810fa>

Jan  8 13:56:39.528 a8 appl[1592]: 1749764.54 SipPacket   Refer-To: <cid:3ac0bec2e9862d4b@127.0.0.1>

Jan  8 13:56:39.528 a8 appl[1592]: 1749764.54 SipPacket   Referred-By: <sip:812000@voiceinitiate.com>

Jan  8 13:56:39.534 a8 appl[1592]: 1749764.54 SipPacket   From: <sip:812000@voiceinitiate.com>;tag=c280f01b2b95aae4 Jan  8 13:56:39.535 a8 appl[1592]: 1749764.54 SipPacket   To: <sip:124758@10.20.40.10>

Jan  8 13:56:39.535 a8 appl[1592]: 1749764.54 SipPacket   Max-Forwards: 70 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

so here is the explanation for the logs shown above:

1)Because the user pressed the merge button, the SX20 endpoint tries to transfer the call to the MCU so it ‘Refers’ the call to the MCU

2)       by sending a refer message to the VCS at Ip address: 10.86.24.90

3)       The SX20 tell the VCS to refer the call to ‘ Refer-To: cid:3ac0bec2e9862d4b@127.0.0.1’; which is basically an incorrect  address. Notice that the ip address of the refer url is actually a loopback address which is not the ip address of the MCU ? So basically the endpoint it trying to refer the call to itself when it does not have multisite capabilities. Even though it needs the MCU to host the conference, the SX20 actually tries to host the call. . A refer to  :3ac0bec2e9862d4b@127.0.0.1 is not the same thing as Refer-To: multiway@voiceinitiate.com so it fails.

4)       See the logs  below for the response that the Video communication server sent back to the SX20 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The VCS responds back basically saying ‘ I don’t know that address’ : We know this because we see can see  a 400 message in the logs.  A 400 message means that the requester sent a bad request with a malformed syntax.

::::::::::::::::::::::::::::::

a8 appl[1592]: 1749764.68 SipPacket I: PacketDump: Proto: SIP, Direction: Incoming, Name: 400 (Refer-To) Unknown SIP URL prefix, CSeq: 101 REFER, RemoteAddress: 10.86.24.90:5061, CallId: 55c9039b4884ec8f0c834997b05cfa10, Time: 1749764678

Jan  8 13:56:39.682 a8 appl[1592]: 1749764.69 SipPacket   SIP/2.0 400 (Refer-To) Unknown SIP URL prefix

:::::::::::::::::::::::::::::::::::::::::::::::

As  can be seen in the logs, this is very abnormal behaviour.when I complained about what I was seeing in the logs to  Cisco, I was informed that this was actually bug CSCug95623. As of today, I believe this bug is not officially published yet.   Hope this has been helpful. All the best


* Hint: Please watch this Video in  HD and full-screen mode or the Video may not be clear.

 

Not too long ago I hit a brick wall while I was configuring the ‘ Recording’ feature of   Cisco’s On-Premises Webex Meeting Server . The problem was that the webex meeting server was saying that it could not find my Network file system. The recording feature actually requires that a Network File System be set-up before users are allowed to record their meetings.  I checked on Cisco’s support site and found out that I was not the only one to have encountered this problem.  Anyway, since I did not find a solution at Cisco.com, I decided to recreate my solution here so that the next person that encounters this problem will have a quicker solution than I had.

Hope this post is  helpful to everyone and maybe even someone who’s blog has been useful to me too.

cheers

Maxwell


This is another attempt to share what I’ve researched in my personal time with other Engineers simply because  I have benefited  a lot  from reading  other people’s blogs also.  My  hope is that this post is useful to someone out there.

Not too long ago, I wrote a post about  the boot process of Cisco jabber for windows which can be found    here.  In continuation of this entry, I thought I’d add share some more  information on how to read the logs that are generated  when  two people who are  on  Cisco Jabbber clients start chatting or sharing screenshots with each other.

Knowing what to expect in these logs could prove very important if a client complains to you that when he sends chat message to someone the message is never delivered.  Anyway, before I start talking too much, lets begin.

Lets starting of by talking about the trace file  that should be  used when looking into the chat messages or picture exchange between tw0 people on jabber clients.   In this regard, the best trace to collect is the XCP Connnecton Manager traces.  It is also advisable to set the trace level to detailed before attempting to collect the trace file. So the question is,  ‘why the XCP Connection manager trace’ ? Well the XCP Connection Manager  helps jabber clients to connect to the IM and Presence Server. It is sort of like an intermediary.

Now that we have that under our belt, the nex thing to know is how to differnciate one chat message for another in a massive pile of traces.

There are several things unique to every chat conversation:

1)   The sender and the receiver’s jabber ID  (JID)

2)  Every message has a Unique ID which is used to identify a message as it is transported across multiple  IM and presence servers en-route to it’s destination or jabber user.

When reading jabber traces, the same rule that applies to phones and call-manager applies here too:  and that is, you will mostly only find the logs for a given jabber client on the server that it is registered to.  This is true for the XCP Connection Manager traces.  What I’m trying to say here is that sometimes you have to collect logs from multiple IM and presence servers but most of the interesting logs will piled up in one server.

Now that we understand that, let talk briefly about the message flow.

For every chat message/ conversation there are several  messages that are exchanged  in the following order.

1) One of the very first messages is the   “IN FROM CLIENT” message. This is the trace line that tells us that someone has sent a message via the XCP Connection Manager  to the IM and presence server using a jabber client.

2)   “OUT TO SERVER” message. This means that the message has now been transferred to the IM and Presence server’s internal processes so that  a destination for the message can be calculatated or derived. It is at this stage that components like the XCP Router kicks in.

3) ” IN FROM SERVER” message. This is the message that  instructs an IM and presence node to deliver a message to a particular XMPP client. i.e jabber.  Please note that if jabber clients are chatting with each other but they are not registered to the same IM and Presence node, you will never see an  “OUT TO SERVER” and an ” IN FROM SERVER” message together on any single node for one conversation going in one direction (i.e from sender to receiver).

4)   Then there is the  “OUT TO CLIENT ” message. This is one of the final messages you will see in the logs when a message is about to be  delivered to  a recipient jabber client.

Ok now that we’ve discussed this matter at length, let take a look at some logs.

In the logs below, user2@voiceinitiate.com sends a message to  User1@voiceinitiate.comand a message ID of 52a46470:00000990:0000010c is assigned to the message

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

[IN FROM CLIENT state:6]: <message id=’uid:52a46470:00000990:0000010c to=’user1@voiceinitiate.com’ type=’chat’><body>hello this is maxwell. i am having a cup of tea so feel free to play your xbox at work</body><thread>connect23313xmlns=’http://jabber.org/protocol/xhtml-im’&gt;hello this is maxwel. i am having a cup of tea so feel free to play your xbox at work

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The Jabber session manager  (cm-1_jsm-1) that is running on server “subcup.voiceinitiate.com”,  informs us that this  message is being sent out to the server’s internal processes for analysis and message delivery

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

cm-1_jsmcp-1.subcup-voiceinitiate-com [OUT TO SERVER(3)]: from=‘user2@voiceinitiate.com/jabber_22990 id=’uid:52a46470:00000990:0000010c to=’user1@voiceinitiate

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

In the  output below, the  Jabber session manager  (cm-1_jsm-1) that is running on server “pubcup.voiceinitiate.com”,  informs us that it has received a request to deliver   message to user1@voiceinitiate.com to user2@voiceinitiate.com

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

cm-1_jsmcp-1.pubcup-voiceinitiate-com [IN FROM SERVER]: from=’user2@voiceinitiate.com/jabber_22990′ id=’uid:52a46470:00000990:0000010c’ to=’user1@voiceinitiate.com’ type=chat’ xml:lang=’en’><body>hello this is maxwell. i am having a cup of tea so feel free to play your xbox at work</body>connect23313xmlns=’http://jabber.org/protocol/xhtml-im’&gt;hello this is maxwel. i am having a cup of tea so feel free to play your xbox at work

 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

In the logs below, we see that the  message is delivered to the recipient jabber client.

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

cm-1_jsmcp-1_xmppd-1 [OUT TO CLIENT]: <message from=’user2@voiceinitiate.com/jabber_22990′ id=’uid:52a46470:00000990:0000010c’ to=’user1@voiceinitiate.com type=chat xml:lang=’en’>hello this is maxwell. i am having a cup of tea so feel free to play your xbox at workconnect23313xmlns=’http://jabber.org/protocol/xhtml-im’&gt;hello this is maxwel. i am having a cup of tea so feel free to play your xbox at work

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Ok so now that we know how text/ chat messages flow within the server(s), how about when we send a screen-shot to someone via jabber? Well  the snapshot below shows me sending a picture from one jabber client to another . I will not show you the full message follow because we already have a general idea of what to expect. However, what  I will show here is the actual traces that showed what was happening in the background as the picture  was being sent.

Let me first of show you the  actual screen-shot  that I took while the picture was being transferred between jabber clients. Pay special attention to the name of the file and see if you can see  that same file name in the the logs below.

jabber

cm-1_jsmcp-1_xmppd-1 [OUT TO CLIENT]: <iq from=’user1@voiceinitiate.com/wbxconnect’ id=’uid:52a45b16:00004d03:000001a7‘ to=user2@voiceinitiate.com/jabber_22990′ type=’set’ xml:lang=’en’>Successfully sent file user1@voiceinitiate.com_20131208_150328.png(13613 bytes)’.</x></jingle></iq>

 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The logs above withe XCP Connections Manager logs. Now the presence Engine logs are even more interesting. Check out the logs below that even tells you were the file was saved on the computer hard-drive.

 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

system.pe.jabber 1308670 INFO ClientEmComponent::onPacket() – PACKET RECEIVED::  <message from=’user1@voiceinitiate.com/wbxconnect’ to=’user2@voiceinitiate.com type=’chat’ xml:lang=’en’> xmlns=’http://webex.com/connect/imcmd’/>Subject&lt;body xmlns=’http://www.w3.org/1999/xhtml’>&lt;img alt=’Screen capture contenteditable=’false’ id=user1@voiceinitiate.com_20131208_150328.png’ name=‘connect_screen_capture src=’file:///C:/Documents%20and%20Settings/user1/My%20Documents/MyJabberFiles/user1@voiceinitiate.com/user1@voiceinitiate.com_20131208_150328.png/></body></html></message>

Anyway, I hope that what has been shared here today will come in handy to someone down the line.

All the best

Maxwell


HINT: If you have soft-phones or Soft Video clients  on your network then this is not for you.  You might want to explore the use of access list instead. 

 

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Catalyst 3850 Configuration samples 

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

This configuration was designed to optimize a network that generates 30% Voip and Video  traffic whilst the remaining is bulk data.  This solution is currently  working perfectly in  an extremely busy network. The brief was that the configuration be kept simple yet effective

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Steps:: 

Create two class maps: The first class maps matches the DSCP  and COS markings of Audio and Video traffic:

class-map rtp_audio_and_video
match dscp af32 af33 cs4 af41 af42 af43 ef
match cos 4 5

class-map signal
description voip signal traffic
match dscp cs3 af31 af32 af33
match cos 3

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

One of the cool features of the 3850 switch is that it allows for the creation of two priority queues. So on this occasion, I placed both  Audio and Video traffic in the first priority queue while placing the VIOP and video signalling traffic in the second priority queue. However, please note that because Video traffic is more burst-y than RTP audio, it is better to place Video traffic in the second priority queue when designing qos for a network where there is an extensive video deployment.

Notice that at the end I just  added the command:  ‘ class call-default’ ? This is the ‘catch-all’ statement that matches any traffic that was not expressly matched by the   class-maps above.

policy-map media_priority
class rtp_audio_and_video
priority level 1 percent 35
class signal
priority level 2 percent 15
class class-default

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The statement below can then be applied to all trunk interfaces. For trunk-groups/ether-channels, you can add the command in both input and output directions.

service-policy output media_priority

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

The following commands were placed on all the access ports. The first command tells the switch to only accept qos markings from  cisco-phones. However, the Cisco 3850 switch provides for  the ability to trust other devices like ,  Cisco Digital Media Player, Cisco TelePresence System, and  IP Video Surveillance Cameras.

The second line tells the Cisco phone to mark all traffic coming form the connected PC with  a COS value of 0.

One thing that should be noted is that by default, the Cisco 3850 switch will trust all QOS markings coming from attached devices so I would advise on using the ‘ trust device’ statements to lock-down  or prevent rouge devices from marking the QOS  values of their traffic too high.

trust device cisco-phone

switchport priority extend cos 0

service-policy output media_priority

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Now that we’ve talked about the configurations, let talk about how to check whether the qos is actually working.:

Looking that output below, you will notice that ‘bytes output’ and ‘total drops’ counters. These are the counters that you need to look out for: issue the command every few seconds and see whether the counters  are increasing.

You will also notice that the class-map counters are all zero. Don’t let this alarm you.  The qos is working; Cisco has just done what they do very well which is to get everyone startled. Ignore that part.  If in doubt, apply an aggressive policing  policy to the traffic that you are matching and see everything grind to a halt 🙂

BRAZIL-3850-STK1#show policy-map interface g1/0/1

GigabitEthernet1/0/1

Service-policy output: media_priority

queue stats for all priority classes:

Queueing

priority level 1

(total drops) 0

(bytes output) 893757086

queue stats for all priority classes:

Queueing

priority level 2

(total drops) 0

(bytes output) 34404961

Class-map: rtp_audio_and_video (match-any)

0 packets

Match:  dscp af32 (28) af33 (30) cs4 (32) af41 (34) af42 (36) af43 (38) ef (46)

0 packets, 0 bytes

5 minute rate 0 bps

Match: cos  5

0 packets, 0 bytes

5 minute rate 0 bps

Match: cos  4  5

0 packets, 0 bytes

5 minute rate 0 bps

Priority: 35% (350000 kbps), burst bytes 8750000,

Priority Level: 1

Class-map: signal (match-any)

0 packets

Match:  dscp cs3 (24) af31 (26) af32 (28) af33 (30)

0 packets, 0 bytes

5 minute rate 0 bps

Match: cos  3

0 packets, 0 bytes

5 minute rate 0 bps

Priority: 15% (150000 kbps), burst bytes 3750000,

Priority Level: 2

Class-map: class-default (match-any)

0 packets

Match: any

0 packets, 0 bytes

5 minute rate 0 bps

(total drops) 0

      (bytes output) 2422713863

Did you notice anything  odd about the Class-maps configurations above? Ok let me ask the question: Are those class-maps a ‘ match-any’ or ‘match-all’ statement?

We all know that if an Engineer  does not expressly configure her  class-map with a  ‘ match-any’ statement,  the class-map  will be set to  a ‘ match-all’ statement. But in the 3850 switches; this is not true.  A quick look at the output of the ‘show policy-map interface g1/0/1’ above will prove this.

Hope you find this helpful.

Cheers

Maxwell


* Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab

* Hint:Please double-click the snapshots to zoom into them. 

This is another interesting case that  I  recently worked on with a colleague of mine.  The topology below represents a high level overview of what the client’s network looks like. The complaint was that users could call local numbers but all international calls were failing to connect.  That got us thinking . . .  our first question was, if local calls are working fine, and international call are not, then what is the difference between both calls

The snap-shot below shows how the sip messages were flowing for the working local calls .
Working call

If you compare the sip messages of the working call above with the non-working international call below, you will see that the difference is that the messages below contains a ‘ 183 session progress message‘.

Failed call

So based on the above, the question that now presents itself is:  why did the Cisco Unified Boarder element send a bye message to the Cisco Call Manager when the remote phone that is out in the cloud was actually ringing? Does this make sense;  Strange isn’t it?

Ok so we know from our experience as Engineers  that when a call fails,  there is normally some sort of error message, failure or cause code of some sort. In the trace output below, you will see that I have actually highlighted the error -code that was in the  ‘ bye‘ message that we saw in the call-flow diagram above.

As we can see,  the sip protocol informs us that the call failed with  a cause of: ‘Reason: Q.850;cause=65‘ . If you do a Google search for cause code 65 will probably find something related to a codec/capability  negotiation issue; which might lead one to think about transcoding. Please have a try for yourself; do a Google search and see what you come up with.

::::::::::::::::::::::::::::::::::::::::::::::::

Sent:
BYE sip:447578332431@192.168.0.10:5060 SIP/2.0
Via: SIP/2.0/UDP 192.168.0.20.8:5060;branch=z9hG4bK8052F20C7
From: <sip:00412256731@192.168.0.20>;tag=FAA7C024-667
To: “maxwell o” <sip:447578332431@192.168.0.10>;tag=342693~12b9673c-9ebc-4b8c-b0db-46b8d985386a-34835506
Date: Mon, 30 Sep 2013 12:44:02 gmt
Call-ID: fa88da80-24917212-28046-2212b0a@192.168.0.10
User-Agent: Cisco-SIPGateway/IOS-15.2.4.M1
Max-Forwards: 70
Timestamp: 1380545045
CSeq: 101 BYE
Reason: Q.850;cause=65
P-RTP-Stat: PS=19,OS=4480,PR=25,OR=4000,PL=0,JI=0,LA=0,DU=34270069
Content-Length: 0

::::::::::::::::::::::::::::::::::::::::::::::::

Did you carry out the Google search, did it show something related to codec?  if so, then I’m pleased to inform you that answer is  incorrect 🙂 lol . . . just joking :-). I have been mislead by so many error codes in the past that I no longer trust them completely.

On a more serious note, lets take a look at what RFC: 3262(click on this link) has to say about this; and I quote:

 If the UAS had placed a session description in any reliable
   provisional response that is unacknowledged when the INVITE is
   accepted, the UAS MUST delay sending the 2xx until the provisional
   response is acknowledged.  Otherwise, the reliability of the 1xx
   cannot be guaranteed, and reliability is needed for proper operation
   of the offer/answer exchange.

::::::::::::::::::::::::::::::::::::::::::::::::

So the question is, what does the above mean? Lets start of by explaining what a ‘provisional response’ is:

A very simplified definition of provisional responses is that ‘ they are 1xx responses within a Sip Dialogue/ conversation’. The snap-shot below provides  a quick glance at examples of Provisional Responses.

(source: http://en.wikipedia.org/wiki/List_of_SIP_response_codes#1xx.E2.80.94Provisional_Responses)

1xxreal

However, the definition above still does not explain what a ‘ reliable   provisional response’ is.

As simplified definition of a ‘ reliable provisional response’ is that :    a provisional response becomes reliable when the Sending User Agent Server (UAS) expects an acknowledgement back from the receiving User agent (UA)

With these definitions under out belt, it is now easy to understand what the RFC copied above was trying to say. The RFC is basically pointing out the importance of receiving an acknowledgement from the remote end when the reliable response mechanism is being used. It can actually stop a call from completing if not handled properly.

Lets take a time-out  to quickly take a look  at the SIP-message-flow the copied below. Did you notice the use  of the PRACK?     :

(Source: ‘ borrowed’ form Cisco )

good call flow

I guess at this point, the Sip-call-flow diagram of the failed international  call makes so much more sense now? I guess we can see the cause of the failure now? Basically, the Cisco Unified boarder element sent a bye message to the call-manager because the call manager never sent an ACK message to the ‘183 session progress message‘ it sent to call-manager.

To be honest, this should not have happened though: The Cisco CUBE algorithm was clearly not following the relevant SIP RFC stipulations. But then again who does? I will not explain why the Cisco CUBE was not following RFC stipulations because this post would become too long and boring.

Anyway, here is what we did to solve the problem:

We modified the ‘ sip profile’ attached to the sip trunk so that it would support the sending of RACK ( Provisional Response Acknowledgement)messages to all  Provisional Response messages coming from cube(i.e  18x sip messages).

Here are the steps:

1) go to cucm admin page

2) find the sip profile that your sip trunk is using.

3) then go to device  > device Setting > SIP Profile

4) enable ‘ send PRACK for all 1xx messages

5) reset the sip trunk

6) make sure your sip dial-peer on the cube that is pointing at cucm has something like this in config line:

dial-peer voice 20 voip
  voice-class sip rel1xx require 100rel

Send PRACK

Well as soon as we did this, all the calls started connecting fine.

However, we were having ringing / ring-back issues on international calls such that when an international call was made, the caller could not hear the remote phone ringing.

If you encounter this in your network, here is what you could do resolve it.

On the cucm:

:::::::::::::::::::::::::::::::::::

1) go to cucm admin page

2) find the sip profile that your sip trunk is using.

3) then go to device  > device Setting > SIP Profile

4) tick the check box for ‘ disable early medial for 180’

5) reset the sip trunk

6 on your Cisco Cube you should have a config on your inbound and outbound dial-peer similar to this(if you found that the service provider was not sending the audio or media in 183 /180 message):

dial-peer voice 20 voip
 voice-class sip block 183 sdp absent
voice-class sip block 180 sdp absent (use this single line with care)

see the diagram below for further details .   .    .

local ringback

Hope you have found this entry entertaining and contributory to your quest for knowledge and your day-to-day battles between Man and Machine.

A special thank you goes out  Mike Jones for his contributions to this case which has now led to this publication.

Mike can be found at uk.linkedin.com/pub/michael-cunliffe-jones/46/6b6/2a6/

All the best folks,

Regards

Maxwell


:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Catalyst 6509  Configuration sample

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Thanks for dropping by again.  I have now added   this configuration sample as a follow up to  ‘ part one’ of this particular post.    This sample was applied a  Cisco 6509 switch stationed in a collapsed core. For clarity, I divided the configurations into three parts.

::::::::::::::::::::::::::::::::::::::::::::::::::

Egress Queuing Configurations

::::::::::::::::::::::::::::::::::::::::::::::::::

policy-map type lan-queuing Egress_1p3q8t
class cos_5
priority
class cos_6_&_7
bandwidth remaining percent 20
queue-buffers ratio 15
random-detect cos-based
random-detect cos 6 percent 98 100
random-detect cos 7 percent 98 100
class cos_2_3_&_4
bandwidth remaining percent 15
queue-buffers ratio 25
random-detect cos-based
random-detect cos 2 percent 80 90
random-detect cos 3 percent 90 100
random-detect cos 4 percent 90 100
class class-default
bandwidth remaining percent 35
queue-buffers ratio 40
random-detect cos-based
random-detect cos 0 percent 70 100
random-detect cos 1 percent 40 70

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Ingress Queuing Configurations

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

policy-map type lan-queuing Ingress_2q8t
class cos_5
bandwidth percent 30
queue-buffers ratio 10
queue-limit cos 5 percent 100
class class-default
bandwidth percent 70
queue-buffers ratio 50
queue-limit cos 1 percent 65
queue-limit cos 2 percent 65
queue-limit cos 0 percent 75
queue-limit cos 4 percent 80
queue-limit cos 3 percent 90
queue-limit cos 6 percent 100
queue-limit cos 7 percent 100

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

These Configurations were applied to every trunk port  to other switches and also to switch-ports connected to Unified Communications servers:

platform qos trust dscp

Service-policy type lan-queuing output Egress_1p3q8t

service-policy type lan-queuing input Ingress_2q8t

 

Hope you’ve found this useful.

Thanks for dropping by.  

 


:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Catalyst 3750 Configuration samples 

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Over the past few weeks, I have been working on a number of quality-of-service (Qos) design and deployment projects so I  thought I’d just spend some time to blog about it since I loved it so much.   In the next few weeks, I will add new entries to this blog for each of the three switch models listed above. Today I will be focusing on the Catalyst 3750.

The Sample configurations below were designed to meet the needs of a branch office comprising of over 500 users.  I will not disclose the nature of the client’s business but it’s sufficient  to say that they generate a massive amount of telephone calls per day.  The client complained that the plague of bad call quality was never too far away from them. They were frustrated by the fact that they could hardly hear the people on the other side of the line and sometimes, the calls would just drop off mid-conversation.

The configurations below is a sample of one of  the configurations that I designed and deployed somewhere in the globe.  I have not added explanations for each configuration line because I thought I’d simply allow anyone to ask a question if they needed more explanation. The intention is to provide a sample config that could possibly be of use to someone out there.

::::::::::::::::::::::::::::::::::::::::::::::::::

Design Considerations:

::::::::::::::::::::::::::::::::::::::::::::::::::

This configuration was designed to optimize a network that generates 30% Voip  traffic whilst the remaining is bulk data.  This solution is currently  working perfectly in  an extremely busy network.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

mls qos

mls qos srr-queue input priority-queue 2 bandwidth 30
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 70 30
mls qos srr-queue input threshold 1 75 90
mls qos srr-queue input threshold 2 90 100
mls qos srr-queue input buffers 85 15

mls qos srr-queue input cos-map queue 1 threshold 1 0 1 2
mls qos srr-queue input cos-map queue 1 threshold 2 3
mls qos srr-queue input cos-map queue 1 threshold 3  6 7
mls qos srr-queue input cos-map queue 2 threshold 2 4

mls qos srr-queue input cos-map queue 2 threshold 3  5

mls qos srr-queue input dscp-map queue 1 threshold 1 0 8 10  12  14 16 18 38

mls qos srr-queue input dscp-map queue 1 threshold 1 20 22 26 28 30 34 36

mls qos srr-queue input dscp-map queue 1 threshold 2 24
mls qos srr-queue input dscp-map queue 1 threshold 3 48 56

mls qos srr-queue input dscp-map queue 2 threshold 2 40 32

mls qos srr-queue input dscp-map queue 2 threshold 3 46

mls qos srr-queue output cos-map queue 1 threshold 3  5

mls qos srr-queue output cos-map queue 1 threshold 2 4
mls qos srr-queue output cos-map queue 2 threshold 3 6 7

mls qos srr-queue output cos-map queue 2 threshold 2 1 2

mls qos srr-queue output cos-map queue 3 threshold 3 3

mls qos srr-queue output cos-map queue 4 threshold 3 0

mls qos srr-queue output dscp-map queue 1 threshold 3 46 40 32

mls qos srr-queue output dscp-map queue 2 threshold 3 48 56

mls qos srr-queue output dscp-map queue 2 threshold 2 8 10 12 14 16 18 20 22

mls qos srr-queue output dscp-map queue 2 threshold 2 26 28 30 34 36 38

mls qos srr-queue output dscp-map queue 3 threshold 3 24

mls qos srr-queue output dscp-map queue 4 threshold 3 0
mls qos queue-set output 1 threshold 1 85 90 100 100
mls qos queue-set output 1 threshold 2 80 90 100 400
mls qos queue-set output 1 threshold 3 100 100 100 100
mls qos queue-set output 1 threshold 4 130 140 100 400
mls qos queue-set output 2 threshold 1 149 149 100 149
mls qos queue-set output 2 threshold 2 118 118 100 235
mls qos queue-set output 2 threshold 3 41 68 100 272
mls qos queue-set output 2 threshold 4 42 72 100 242

mls qos queue-set output 1 buffers 10 20 10 60
mls qos queue-set output 2 buffers 16 6 17 61

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

These Configuration were applied to every switch-port that was connected to a Cisco phone:

mls qos trust cos

switchport priority extend cos 0

mls qos trust device cisco-phone

priority-queue out

srr-queue bandwidth share 30 15 10 45

queue-set 1

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

These Configuration were applied to every trunk port.

mls qos trust cos

priority-queue out

srr-queue bandwidth share 30 15 10 45

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

Hope you’ve found this useful.

Thanks for dropping by.  

 


* Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab

* Hint:Please double-click the snapshots to zoom into them. 

This is one of the most intriguing cases I have every dealt with:   This client has Cisco Unity connection 8.6x integrated with a Microsoft Forest containing  over ten Microsoft server 2003 and 2008 domains controllers spread out across multiple sites in Europe and the Americas.

The client complained that all of a sudden, when new users were created in Active directory, they would no longer show up in Cisco unity connection . I asked the client what had  changed prior to the problem and I was informed that nobody had touched the Cisco servers except for  the fact that their Group policies had been recently been re-written within active-directory.

I logged unto Cisco unity connection and tried do an import of users and nothing happened; no error messages were displayed on the webpage and no  newly created users from Active-Directory were imported.

I made up my mind that I would have to  pull out the ‘ Big-Guns’:

Stage one 

1)    I Installed Wireshark on a  PC and started capturing packets

Stage Two

On the same  PC, I also installed Softerra LDAP Browser. softerra LDAP Browser is a tool that you simple put the username, password and  IP-address of your LDAP server into and it will show you everything in your Domain. This was going to be the way that I would use to simply test the username and password that Cisco unity connection was using.   If you need this tool, here is the link to it: http://www.ldapbrowser.com/download.htm

Here is snapshot  of the tool in action:

LDAP-Browser

Stage Three

After I entered the username,password and IP-address of the LDAP server, I started seeing errors in the Wireshark. Here is a screen-shot of the errors below:

LDAP bind failure

The username and password that I was using to authenticated was the same one that the Cisco unit Connection server was using so from the wireshark capture, it was clear that the reason the unity connection server was unable to import users from Active Directory was because it did not even make it pass authentication let-alone pulling user database.

If you look at the error message in the capture above, it say: ‘ In order to perform this operation, a successful bind must be completed on the connection’  

The term ‘bind’ means authentication. So in ordinary English, the error message that can be seen in the LDAP message coming back from the LDAP server  is basically saying: in order to search the LDAP user database, you must first authenticate successfully.

I double checked with  the ‘DirSync’ service traces on unity connection and I found exactly the same error message.

Stage Four

With this evidence, I told the client to speak to his Server team; who in tern, contacted Microsoft Directly. Shortly after the Microsoft team arrived on the scene, they informed me that they did not share my view that the problem was on the Microsoft server side.  They stated that since other  users and servers were authenticating against LDAP without any problems, the LDAP server was fine.

Stage Five:

I knew I was talking to skilled and experienced Engineers so I was not quick to doubt them. I pulled out my wireshark again and investigated how the other non-cisco servers were managing to authenticate without any problems. what I found was that Cisco Unity Connection was using a method of authentication called ‘ Simple Bind‘ while the other servers were not.  Simple bind authentication method is when  usernames and passwords are sent in clear text: I will share how I got to know this later in this post.

I went back to Microsoft and informed them that they  needed to permit simple bind for the Cisco Unity Connection server because that is how Unity connection operates. And that if they wanted security, they would have to use ‘ transport-Layer-security (TLS)’ because that is what Unity Connection supports.

Stage Six: 

In order to test my theory, Microsoft then decided to enable Anonymous Bind for test purposes only. Anonymous bind is when a client can authenticate to the server without username or password.

 

Stage Seven

After we made the change above, the Cisco unity Connection still could not import users from Microsoft Active directory. Again, the Microsoft Team told me that they did not share my view . However, I reminded them that  I  asked for ‘simple bind’ to be allowed on the network not ‘ anonymous bind’ so they continued to work with me.

I was really starting to look really  bad.  I pulled out wireshark again and captured the packets being exchanged between the Cisco Unity Connection server and Microsoft LDAP Server and noticed that something had changed since anonymous bind was allowed;  this is what I found:

LDAP_no such object.

Here is the interpretation of the logs above:

1)  In the very first line, we see that Cisco Unity Connection server at Ip-address 172.16.6.33 sends an LDAP simple bind request to the LDAP server at IP-address  172.16.6.5. We know that we are using simple bind authentication because can see the word ‘ simple‘ at the very end of the line.

2)  Looking at the ‘ source’ and ‘destination’  fields In the second line of the capture, we can see that the LDAP server sends a message back to the unity connection server stating that authentication had successfully completed.

3)  In the third line, we see  the Unity connection server trying to find all the users in the database by sending a search request to the LDAP sever and in the fourth line we see that the server responds with a result of zero; meaning that no result was provided to the Cisco unity connection server.

4)  Going further down the line, we see that the unity connection server continues trying until the 20th line when the LdAP Server responds back with NameErr  0000208D, Problem 2001 (No_object)

Stage Eight

I decided to collect traces  from Unity Connection  so I turned on detailed traces, performed a full LDAP system sync on unity connection and collected Cisco Dir-sync service traces from the unity connection server. Here is what I  found:

DAPSync(cca4b920-9361-9535-8d80-ece481332007)[getInvocationId] caught exception … [LDAP: error code 32 – 0000208D: NameErr: DSID-03151F00, problem 2001 (NO_OBJECT)

 According to the following Microsoft website, LDAP error code 32 means:  ‘The user has insufficient access rights: http://support.microsoft.com/kb/218185

Stage nine

I Informed  the Server Admins and  Microsoft team about the new error messages I was getting from their Microsoft servers.  I asked the team to explain why their server was not happy with my request since the account I was using to authenticate was an admin user. In response, they informed me that the server was treating my Simple bind Request as an Anonymous bind request and since Anonymous bind request were not permitted to search the database, my request were being dropped.  To remedy this, the following steps were applied to the Domain Controller.

  1. Open regedit.exe on a domain controller
  2. Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders
  3. In the right window pane double-click on SecurityProviders
  4. Add pwdssp.dll to the end of the Value data field which should be separated by a comma after the last entry.  For example:

msapsspc.dll, schannel.dll, digest.dll, msnsspc.dll, pwdssp.dll

5.  Reboot the DC: This was done on all the domain controllers on Windows 2008 and above and a reboot was required

Stage Ten

After the changes above were applied, everything started working fine  lol 🙂 . . . and all along, everyone was blaming me but thanks to wireshark, I was able to prove to the remote Microsoft Server team that the problem was from them.  Long live Wireshark. 🙂 and Mr Remi A @silvercleric for his contribution to this case.

%d bloggers like this: