#76 2009-04-07 09:55:47

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Quick update from me on this - So far we know about some udates last week by Adobe and some upcoming updates, but there is still no clear indicator as to the cause of the problems or explination as to why they have become so much worse than before the SP2 release.  We tested late last week with Adobe and weren't able to get good data, so we are testing again tomorrow during a live event.  Hopefully we will get something more substantial this time.

So far things suck and are only getting worse.  Good news is Adobe is working on it.  Bad news is what they are finding and fixing isn't helping.

-Chris

**_et_**

Re: VOIP audio drops increase after SP2

Hi folks - apologies for my absence in the forum recently.  I was out of the office for a couple weeks with my family at my brother's wedding, and got back earlier this week.

First, just to reiterate our commitment to this issue:  the team looking at the VoIP issues and reports is fully engaged with your feedback and works on this every single day.  In general customers have a great experience with VoIP, but if you're reading this you probably have run into issues.

Second, for those who experience issues, there are many variables that are in play.  As a result, our approach is to pursue several solutions concurrently, and deploy each one independently to gauge whether the improvements observed during testing hold up in the context of the networks and usage patterns of those who see problems.

Here are the two changes that have occurred or are in progress:

1. For the NA3 cluster only, last Friday night (4/3) a change was deployed in the network during a scheduled maintenance window.  This change was to route traffic away from a switch that was dropping frames for a subset of the NA3 cluster.  The result should be more consistent and increased throughput as well as reduced latency for all traffic to/from NA3, including VoIP traffic.  If your account is on NA3 you may have already experienced this improvement over the last week.

2. For all clusters, tonight (4/10), the media streaming configurations are being tuned in a scheduled maintenance window (happening right now) to prioritize VoIP and provide our support and engineering teams with additional logging and debugging information to further analyze reported issues.  We will be very interested to hear if you observe an improvement after tonight's maintenance (no pun intended :)

Lastly, there are other changes in the works, but as I mentioned, we're taking this one step at a time, continuing to gather data, and working on not only short term improvements (e.g. like the above changes) but long term improvements as well - things that are designed to dramatically improve VoIP in the next major rev of the product.

I am very interested in your experience and your feedback, please keep it coming.

thanks,
Eric

p.s. For those who were interested in trying out another cluster for comparison, it's still possible but I no longer believe it's necessary given the 4/3 and 4/10 changes.  I am happy to oblige the previous requests if your current account still exhibits issues, so that we can further diagnose the problem.  Try VoIP again after tonight's maintenance and let me know.

**_JohnK_**

Re: VOIP audio drops increase after SP2

Eric,

Thanks for the update.

Does this also apply for EMEA accounts? I have received Maintenance Announcement for 4/11 and again 4/18. Actually I received the 4/18 announcement e-mail 3 hours before the 4/11 was scheduled.

Does the 4/18 Maintenance Announcement have any relation to this topic?

John K.

**_et_**

Re: VOIP audio drops increase after SP2

@JohnK - yes, the 4/11 maintenance included EMEA.  For 4/18 we're just planning ahead and notifying early; the 4/18 maintenance is specific to EMEA and not related to this topic.

Thanks for asking.  And let me know if you notice any improvement!

Eric

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Eric - First, thank you for the update, I am happy to say that I have seen a perceived improvement since the 4/3 update, but I cannot say how much of an improvement it is.  I have had a mostly successful meeting with relatively few drops, but still more drops than before the SP2 update.

That being said, I think you are stepping out on a big limb saying that customers have had a great experience with VoIP in general.  I personally have had a number of complaints for over a year, most notably when I started this thread after the SP2 release.  I have also spoken to many others who have had problems much longer than I have.  When Adobe resorts to things such as recommending to not use enhanced audio (which should improve the audio) in order to prevent audio problems, you cannot being to tell me that customers have a great experience.  I would say customers had better experience before Adobe acquired the product.  After Adobe broke things with ACP 7 (Adobe's first major release), there has been nothing but problem after problem with the product which especially includes VoIP.  Why do you think so many people rely on telephony, Skype, and other sources for audio?  Not what I call a great experience in general.

With that said, Adobe has made progress in correcting many of the issues it introduced with ACP 7.  Based off of recent updates, maybe you are starting to figure things out with VoIP (the concern of this thread), but I am not crossing my fingers.  I have spoken with support time after time after time, and I understand that this is a large issue that Adobe is having trouble wrapping its hands around.  Adobe hasn't identified the cause yet (I don't think there is a single smoking gun), but has identified many other problems you have since corrected.  If nothing else, that is an improvement for the product (for hosted customers at least) and is appreciated.  This is really the first promising thing I have seen come from Adobe regarding support of Connect since I started using it a few years ago.

So I ask you, what does all of this mean for us, the customer?  New changes for the next major rev of the product that should improve VoIP (and hopefully many of the other bugs the product has)?  How can we trust that these are improvements and not just more headaches waiting to happen?  Is this coming with version 7.5 in September, or version 8 some time later?  Maybe things are looking a little better, but is it enough to string customers along until the next rev of the product? 

I am harsh on Adobe, maybe more so than I ought to be, but Adobe has earned my harsh words time and time again.  For once, I have seen action on an issue other than me asking for an update, for which I am appreciative.  I have a strong opinion it is due in part to this long, vocal thread of complaints and encouraging others to report all VoIP problems to help find a solution.  I have done my part to test and provide feedback to Adobe, and for once I have seen Adobe try to be part of a solution and not add to the problem.  I just am tired of justifying to my clients why I use something that causes so many problems, and I am concerned they may not be clients very much longer.  I don't know that Adobe shares that concern with me.

Again, thank you Eric for the updates.  I can't speak for everyone, but I do appreciate the effort that Adobe is putting into this VoIP issue, and I hope it is resolved soon.  I just can't say that I believe a word you are saying about customer experience, especially since I have never heard any customer state what you claim.  Not to mention what I have heard people from Adobe say about VoIP that contradicts your statement.  I look forward to the day that no one has to update this thread any longer (especially me).

Thanks
Chris

**_et_**

Re: VOIP audio drops increase after SP2

Chris,

Thanks for sharing your thoughts--and for originating this thread--I can tell this has been a particular issue for you and I'm glad you continue to engage us.

Also, I'm glad you've seen an improvement since the 4/3 change--you should have, since you mentioned your account is on NA3.  I'm also curious to see if you've seen an additional improvement since Saturday (after the 4/10 change).  If so, we're still headed the right direction, at least for the particular issue you're seeing.  I'm interested either way, so please try it out and let us know.

As of the 4/10 change we have the ability to gather additional data regarding streaming connections (like VoIP).  For anyone who is still seeing an issue, this data will help us narrow down the issue.  As I mentioned earlier, there are many variables in play, so the easier we make it to troubleshoot an individual case, the easier it is to reproduce and resolve it.

On the release schedule, we'll continue to make critical updates to C7 as needed, and as we make these changes we'll continue to provide updates and solicit feedback.  In your particular case, I'd like to follow up directly and understand your situation a bit better.  While I don't think an alternate cluster test is as useful as it might have been prior to the NA3 change, it may still give us some information.

thanks
Eric

**_bbjawer_**

Re: VOIP audio drops increase after SP2

Chris, our experience with ACP VoIP is exactly along the lines you mention in your latest post.  We also feel that while Adobe has tried to resolve these VoIP issues, they have not yet gotten their arms around the problem.  We have been using hosted ACP for the past 7 months now (V7.0) and have always had VoIP issues from the beginning, and continue to have them.  I hope the 4/10 fix  helps improve VoIP quality, but at a minimum I just hope it fixes what a previous hot fix broke (causing visual and audio to go out of synch in recordings of VoIP sessions).  It is clear that VoIP problems are affecting many ACP customers, not just one or two.   Thanks for originating the thread and for continuing this discussion.  In the end, if Adobe can get this thing fixed, it will help everyone involved.

**_roysdenc_**

Re: VOIP audio drops increase after SP2

New update - had a session this morning and things were no better than they were before the 4/3 update.  It is possible that any ground made with the 4/3 update was lost with the 4/10 update. 

2 people in the room (me and the presenter), slides only, no screen share, 10 minute presentation with multiple 5+ sec drops.

back to pen and paper...

Ugh
chris

**_jcooper9099_**

Re: VOIP audio drops increase after SP2

I have a question for adobe. What is the maximum jitter threshold on your VoIP network?

**_eomedia_**

Re: VOIP audio drops increase after SP2

I thought I'd post my two cents here in that we are a conferencing/registration company that integrates with Adobe and have several clients that do quite large events with us and Adobe (up to 1500 people in a single event).

In a majority of our events we utilize our own audio technology with Connect and not their VOIP, however, on several we have used the VOIP and our audio in a blended environment so we do have some experience with the VOIP.

In our experience we've found that the VOIP experience is no better or worse than a great majority of the VOIP offerings that are used in conferencing.  On one of our recent blended events the VOIP acted up at the start of the event with some bad feedback (presenter had audio speakers and microphone open - user error)  but after that the VOIP was as flawless as our audio.  We've also experienced delays, echo's and audio drop outs from time to time.

VOIP is a great technology but IMHO it has not yet reached the level of reliability and quality of traditional telephony, although I don't expect that time to be too far down the road.

The price of VOIP is attractive and in many situations is the best solution and is what we recommend to our clients, if on the other hand they are charging for their events or need an end user experience where they need a better guarantee we suggest more traditional audio as the cost has come down to exceedingly affordable prices especially when compared to just a few years ago.

If VOIP is an absolute requirement I have a feeling that no matter who the provider is one is likely to experience some issues even if they are different from the current ones.  I am a Vonage home user and have been for years and in general their quality has been excellent, but even today there are times when I experience issues that I never get with my traditional land line.

Just another opinion to consider is all.  Thanks.

**_jcooper9099_**

Re: VOIP audio drops increase after SP2

What do you mean by "bended"? The only way that I have found to acheive this is to have a user with two headseats or to use a that-2 device.

Are you doing something different?

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Thanks eomedia

The main thing that we are running into with VOIP is that when Adobe released SP2 in Feb, the quality went from acceptable to often unusable.  One thing that is weird is it seems mainly issolated to hosted accounts on the na3 server.  Adobe is working to correct the issue, but it seems like they are a long way from figuring out based on the performance I have seen as recently as today.

-Chris

**_bbjawer_**

Re: VOIP audio drops increase after SP2

eomedia,

Our company has used (or uses) several webinar and virtual classroom tools (Interwise, Centra, Elluminate, GoTo Webinar, Webex) over the last few years.  EACH one of these have had better quality VoIP than we have experienced with ACP.  With all the other products, we have used VoIP as our main audio delivery vehicle.  Note: When I talk about quality in this context, I am talking about stability - few pauses, few loss of words, etc.  ACP's VoIP voice quality is excellent and is a match for any of the other products I've named above.  However, as far as VoIP stability - ACP's VoIP (at least on NA3 and NA4) has consistently been poor - to the point that we now have to run most of our sessions using the telephone bridge.  Even when there were problems with the other products' VoIP, it never got so bad that we had to switch all our sessions from VoIP to telephone like we have done with ACP.

In short, there are problems with ACP's VoIP that we never experienced with any of the other products we have used.

**_eomedia_**

Re: VOIP audio drops increase after SP2

Well I certainly did not mean to stir up a sore point for anyone, just wanted to drop an opinion.  :)

jcooper9099:  You can do that via the that-2 device or through two unique lines as indicated, and we've done it via the that-2 device.  We have also utilized a pod to dial out to our bridge and tie the audio and VoIP together.

roysdenc:  I can certainly understand that type of frustration, we run on the na3 cluster ourselves and have generally had a decent experience to date.  That said, we RARELY use the VoIP on those situations as we utilize our own audio technology on these events.  Our sampling rate is likely very low compared to your usage of the VoIP and perhaps if we used more VoIP we would also experience those issues, I guess we're lucky that VoIP is not a requirement in most cases for our clients.

bbjawer:  Absolutely no way to argue that with you, personal experience with a product will absolutely form an opinion of the different technologies.  I've used all but one of the tools you mentioned above and have had different issues with those technologies regarding the VoIP like I have with Adobe's, no better or worse in my personal use cases.    It's kind of like reading reviews online where 5 people will give the product 5 stars and 3 people will give it 1 star, it all depends on your experience.

My only suggestion is that when looking at the available products one has to look at all the features of the product and determine what best serves your needs, and if VoIP is the only problem you have with Connect there are other options to replace the audio/VoIP.  If VoIP is essential to your value proposition and one continues to have issues ongoing then by all means look into other solutions, my point is just to keep in mind that you may encounter VoIP issues with these companies as well at this point in time.  I'll bet we could find postings about VoIP issues with any number of solutions.  This doesn't mean that I don't think Adobe should be working full speed to fix any issues they have and we all know they do, just that one be careful in choosing the right solution.

**_roysdenc_**

Re: VOIP audio drops increase after SP2

eomedia - Thanks for your post, and if anyone has stirred up a sore point, it would be me with this thread.  I like your thoughts and your opinions are welcome, as that is how we learn from each other.  Everyone's situation is uniuqe, and I can honestly say that I have not found another solution that can do what Adobe does with Connect based on the multiple ways it is used here at my company.  We have found other solutions, but they are not as acceptable.  I am actually about to go into a meeting to discuss how we want to move forward for one group due largely in part to this VoIP issue.

Again, any thoughts, opinions, feedback are always welcome here.

-Chris

**_et_**

Re: VOIP audio drops increase after SP2

Hi,

@roysdenc:  Regarding the question regarding whether the 4/10 change invalidated or reversed the 4/3 change.  The quick answer is no -- the two changes were very different -- the 4/3 change was a hardware switch specific to NA3, and the 4/10 change was an updated code configuration for all clusters.  The latter is providing us with increased data regarding streaming performance in order to help troubleshoot specific reports to see where the problem originates so it can be isolated.  Our support teams have already started using this data.

@jcooper9099:  Support has been working on tunables in individual cases to see if they yield improvements.  If they're not already working with you let me know and we should be able to see if they help in your case.

@bbjawer:  I will check internally on the out-of-sync issue you mention and see what update I can provide.  I don't believe the 4/10 update included changes in this area.

Eric

**_bbjawer_**

Re: VOIP audio drops increase after SP2

Eric, according our TAM, the 4/10 changes addressed those recording problems.  And, in fact, we have run a few tests since 4/10, and those particular recording issues we had been experiencing seem to be fixed.  Thanks.

**_roysdenc_**

Re: VOIP audio drops increase after SP2

I need to update as well.  After the 4/10 update we still saw the same problems, little or no improvement.  There have been additional changes/updates since and we have seen drastic improvement starting with the end of last week.

Since last Friday when some changes were made to my account by Adobe, we have not seen any issue.  We will contiue to monitor to determine if this a fluke or not.

In anycase, thanks for the improvement we are currently seeing.

-Chris

**_roysdenc_**

Re: VOIP audio drops increase after SP2

So, the improvements were short lived.  Throughout the week things degraded with VoIP, but at least not to the point they were the week before.  Still collecting data and working with support, so maybe one day things will be acceptable again.

-Chris

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Still getting the same reports of bad audio from my users, as well as experiencing it myself.  As of this morning, Adobe will be undoing some changes to my account that we thought made improvments because I don't think it helped in the end.  We will see what tomorrow brings.

In the mean time, I have a trial account on another cluster that I am going to do testing on to see what the experience is there.

-Chris

Last edited by **_roysdenc_** (2009-04-28 15:59:59)

**_Shelley_**

Re: VOIP audio drops increase after SP2

In our 90-minute live broadcast last Thursday, we had about 100 participants. The VOIP seldom cut out this time, but we did experience latency problems that made the Q&A less effective than I'd like. The long lapse between asking and answering a question made ya wanna go, "Hello, are you THERE?"

Very annoying.

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Shelley - Remind me, are you on the na3 cluster? 

-chris

**_Shelley_**

Re: VOIP audio drops increase after SP2

Chris, we're on the NA3 servers.

**_roysdenc_**

Re: VOIP audio drops increase after SP2

Interesting.  I wonder if some systems on the na3 cluster are working properly and others having issues.  This could explain why we saw good results some days and poor results other days last week.  I know this is something support is looking at for me, so hopefully we will at least know that answer soon.

-Chris

**_mschwachter_**

Re: VOIP audio drops increase after SP2

My last meeting was a disaster with both a phone bridge and VOIP.  We cannot expect those across the world to stay in a lengthy meeting over the telephone.  This problem will destroy this product!  Fix it please Adobe.