VoIP Tech Chat

Patrick and Fred Chat… sometimes about VoIP

Amazon EC2 SIP Brute Force Attacks on Rise

49 comments

Attacks from the cloud.

Update #1: 12 APR 2010. Amazon NOC’s response.
Update #2: 12 APR 2010. Amazon Statement.
Update #3: 13 APR 2010. Amazon Response.

Complaints of rampant SIP Brute Force Attacks coming from servers with Amazon EC2 IP Addresses cause many admins to simply drop all Amazon EC2 traffic. Generally, SIP brute force attacks attempt to register various peer names to a system and/or attempt to guess passwords of known/guesses peers or endpoints.

The complaints mentioned this weekend show an excessive amount of traffic; with some providers claiming 6GB of traffic dedicated to such attacks. Since we ourselves received an attack from an Amazon hosted server, we also reported and complained to the Amazon NOC/Abuse depts. As of this posting, no response or acknowledgement has been received from Amazon. The response from Amazon is below.

There are various techniques to assist with minimizing DDoS and Brute Force attacks, such as limiting access via the public internet, using strong passwords, not mapping extension name to peer/endpoint name, limiting simultaneous calls, and aggressively monitoring usage. Automatic blocking of abusive IP’s (fail2ban, blockhosts, etc.) can also assist with minimizing damage.

Update #1: 12 APR 2010. “Response” from Amazon’s NOC

So when this happened, I submitted a report to Amazon complaining of the attack. The report was sent to their abuse and noc mails and contained the standard abuse report, including their host, my host, the protocol, ports, and description of activity; as well as a sample log.

About 48 hours later, they sent this as a response:

From: Amazon.com <ec2-abuse@amazon.com>
Subject: Your Amazon EC2 Inquiry
Date: April 12, 2010 7:31:59 AM EDT
To: Fred Posner

Hello.

Thank you for contacting Amazon Web Services. We take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use.

Because Amazon EC2 Public IP addresses may change ownership frequently, without additional information we will be unable to identify the correct owner of the IP address for the period of time in question.

So that we can process your report and identify the actual customer in question, we require the following information. Please note that we will not open attachments under any circumstance

  • Source IP
  • Destination IP (your IP)
  • Destination Port and Protocol
  • Accurate Date, Time and *Time Zone* of activity
  • Intensity and frequency of activity in short log extracts, no larger than 4KB
  • Your contact details (phone and email)

For a faster response, please file your report using the AWS Abuse form at the link below:

https://www.amazon.com/gp/html-forms-controller/AWSAbuse/

We appreciate your help in providing the necessary information requested.

Best regards,

-EC2 Abuse Team

So, their laziness aside (since the report to them included ALL the information requested), I filled out the form– which of course failed with an unknown error. (below)

For people interested in moving to Amazon’s cloud, this is a good example of the quality of people conducting your network administration. I’ve sent another response by email. Others have blocked the entire EC2 IP range and have requested their upstream providers do the same.

So my question to you… What would your next step be?

Update #2: 12 APR 2010. Amazon Statement.

Following a request for interview/statement, VoIP Tech Chat received the following communication from Amazon’s Public Relations Manager:

Hello Fred and thank you for contacting us.  Over the weekend, we received a report of a suspicious account and began an investigation.  Our normal process is to connect the two involved parties to give them an opportunity to talk in case the abuse is not malicious but is simply heavy traffic from a legitimate customer.  If that is not successful, we then move to isolate the traffic from the abusing party.  Normally this process works quite well for situations our customers have encountered, however this incident has highlighted the need for an escalation process to address potentially malicious attacks more quickly.  Additionally, we are working on quickly putting better protections and processes in place to better guard against unwanted SIP traffic.  We take the security of our customers and our quality of service very seriously, and will  continue to work to improve our processes and services for customers.

Thanks

Kay Kinton
Public Relations Manager
Amazon Web Services

I’ve replied to again ask for an interview and will update if a response is received. The statement states this was over a weekend, however doesn’t address that the attacks continued today. It also states a “report” was received, but there were many reports submitted. That being said, at least they responded.

Update #3: 13 APR 2010. Amazon Response. Decline of Interview.

After Kay Kinton’s statement, I asked her for a phone interview.

Sent: Monday, April 12, 2010 2:00 PM
To: Kinton, Kay
Subject: Re: Amazon Web Services

Kay,

Thank you for your statement. I would like to interview you about this for VoIP Tech Chat… it would be an over-the-phone interview and would be for 5 minutes or however much longer you would like.

—fred

Kay’s response was quick, and to the point:

What else can I tell you Fred?

I truly dislike email for interviews for no other reason that not getting the tone of that response. Did Kay mean that as “Sure, great! What else can I help you with?” Or, was it more along the lines of, “I answered you. What now?”

Giving her the benefit of the doubt, I replied:

Date: April 12, 2010 5:24:14 PM EDT
To: “Kinton, Kay”
Subject: Re: Amazon Web Services

We would like to interview on this. I thank you for the statement, however I have additional questions:

I know of 12 complaints since Saturday (from different reporters) that were submitted regarding SIP attacks from EC2 to outside systems. How many complaints did you receive since Saturday?

I know attacks continued today and may even be ongoing. There were attacks as of 1pm EST hitting systems with over 640K of data. Are you still seeing attacks? How many hosts were identified?

Were the attacks submitted from one customer/client of yours or many?

Those are my initial questions, however I do request a phone interview rather than email. I find them much easier to exchange information as well as generally a better expressive forum for an interview.

—fred

Good thing I didn’t hold my breath. The next day, after not receiving a response, I called Kay several times and emailed her for an update. Her response via email:

Hello Fred. We believe that we’ve identified and shut down the illegal activity and are closing the loop with customers. We’d certainly be interested in hearing of the cases you refer to below so we can follow up.

I tried reaching out to her but have not had responses. Which leaves me with this…

Her response did not answer my question and I certainly have no basis to believe that Amazon is currently taking any interest in this matter. They’ve told us prior that they cannot pinpoint IP to timeframe as well as that during an attack, they’d try to mediate between parties rather than actually stopping the attack in progress (to give them an opportunity to talk). Sadly… when I’m being flooded, I want the flood to stop. Afterwards, I’ll be glad to talk. But I digress…

Since Kay did not answer any of the additional questions we asked, but did state that she’d be interested in hearing about the other cases, we will encourage anyone with information or feelings about this issue to contact Kay Kinton directly:

Kay Kinton
kinton@amazon.com
Public Relations Manager
Amazon Web Services
Phone: 206-266-8387

For More Information:

Report your error with our form. Fail.

Written by Fred

April 11th, 2010 at 4:14 pm

Posted in VoIP

Tagged with , , ,

«
»

49 Responses to 'Amazon EC2 SIP Brute Force Attacks on Rise'

Subscribe to comments with RSS or TrackBack to 'Amazon EC2 SIP Brute Force Attacks on Rise'.

  1. [...] This post was mentioned on Twitter by .e4 Technologies, Fred Posner, Team Forrest, asteriskbot, topsy_top20k_en and others. topsy_top20k_en said: blogged on VoIP Tech Chat: Amazon EC2 SIP Brute Force Attacks on Rise. http://bit.ly/ec2sipattack [...]

    Tweets that mention Amazon EC2 SIP Brute Force Attacks on Rise | VoIP Tech Chat -- Topsy.com

    11 Apr 10 at 4:22 pm

  2. Perhaps Amazon should be held financially accountable.
    That may motivate them to deal with these attacks in a timely fashion.

    Paul

    12 Apr 10 at 8:23 am

  3. Great idea — Especially for those people getting constantly hit with a good amount of traffic.

    Fred

    12 Apr 10 at 8:39 am

  4. As a service provider, it sounds like maybe:

    1) humans on the other end expect unrealistically consistent inputs (either a hand-formatted email, or a form submission), and/or

    2) there’s not an acknowledgement that folks reporting abuse (or in practice, the small % who cluefully report real abuse) are doing AWS a favor, and that AWS needs to absorb the burden.

    #2 is a superset of #1, but harder to fix.

    Credit to Amazon for at least requesting the right info. We’ve submitted a couple EC2 abuse reports (unrelated to SIP brute forcing) and haven’t had issues, but I also can’t verify that they were followed up on.

    Related to #2: abuse@ is too often a black hole. For most service providers, third-party abuse recipients are one of very few constituents who can’t track issues. A ticket number is useless without self-service visibility or a resolution.

    Troy

    12 Apr 10 at 9:26 am

  5. Troy,

    If the attack was stopped, that would be great. Other than that… no email response and continued attacks is unacceptable. I don’t care if responses are human or not… but if you have an attack in progress that continues for a long period without response, acknowledgment, or prevention from the host network, that’s a problem.

    Fred

    Fred

    12 Apr 10 at 9:42 am

  6. At this point we have asked our upstream provider to block at their core or border routers. Not likely to happen but one can hope.

    Anyone try actually calling Amazon?

    -paul in NV

    Paul

    12 Apr 10 at 10:05 am

  7. Paul,

    I called them at their 206-266-4064 number and left a message (it’s voicemail only). I also called the Law Enforcement line at 206-266-1722 but that is voicemail only (I didn’t leave a message). I called their CS line at 800-201-7575 and after 40 minutes of hold gave up and ended the call.

    Fred

    12 Apr 10 at 10:11 am

  8. Just cancelled my EC2 account. When Amazon gets enough of those, they will quit with the run-around.

    Ward Mundy

    12 Apr 10 at 12:53 pm

  9. Our Asterisk suffered from these attacks a while ago , reported it to Amazon along with all the details but nothing really came of it.

    To prevent any further problems I installed fail2ban and made sure all my extensions have long strong passwords.

    Not had a problem since.

    Mark Waters

    12 Apr 10 at 3:33 pm

  10. [...] Attacks from Amazon servers Link 1 [...]

    iSnick Blog » Blog Archive » Clouds Are Turning Grey

    12 Apr 10 at 5:50 pm

  11. Interesting… RT @fredposner: @danyork what's your take on the SIP attacks from EC2 (still continuing). http://bit.ly/ec2sipattack

    Dan York

    12 Apr 10 at 6:25 pm

  12. [...] http://www.voiptechchat.com/voip/457/amazon-ec2-sip-brute-force-attacks-on-rise/ [...]

    Recent SIP Attacks from Amazon EC2 « Asterisk, and other worldly endeavours.

    12 Apr 10 at 7:28 pm

  13. Amazon EC2 SIP Brute Force Attacks on Rise | VoIP Tech Chat http://bit.ly/9pEnpD < #AWS should be able to provide ownership of IPs @ time

    Steve Chambers

    12 Apr 10 at 11:23 pm

  14. Amazon EC2 team doing nothing about fraudulent use of their resources: http://bit.ly/ec2sipattack – Their report form didn't even work.

    Jazz blues wine voip

    13 Apr 10 at 6:35 am

  15. A listing of the source ip ranges would be most helpful.

    spenser

    13 Apr 10 at 1:53 pm

  16. @spenser:

    This is from the mailing list:

    216.182.224.0-216.182.239.255
    72.44.32.0-72.44.63.255
    67.202.0.0-67.202.63.255
    75.101.128.0-75.101.255.255
    174.129.0.0-174.129.255.255
    204.236.192.0-204.236.255.255
    184.73.0.0-184.73.255.255
    216.236.128.0-216.236.191.255
    184.72.0.0-184.72.63.255
    79.125.0.0-79.125.127.255

    http://lists.digium.com/pipermail/asterisk-users/2010-April/247123.html

    Fred

    14 Apr 10 at 6:59 am

  17. Interesting write-up of AWS-sourced attacks against SIP targets over the weekend: http://tinyurl.com/y2t7g62 < challenges of an operator

    Christofer Hoff

    17 Apr 10 at 12:12 am

  18. RT @Beaker: Interesting write-up of AWS-sourced attacks against SIP targets over the weekend: http://tinyurl.com/y2t7g62

    George V. Hulme

    17 Apr 10 at 12:17 am

  19. RT @Beaker: Interesting write-up of AWS-sourced attacks against SIP targets: http://tinyurl.com/y2t7g62 < challenges of an operator<-ouch

    Vanessa Alvarez

    17 Apr 10 at 12:21 am

  20. RT @Beaker: Interesting write-up of AWS-sourced attacks against SIP targets over the weekend: http://tinyurl.com/y2t7g62 < challenges …

    George Reese

    17 Apr 10 at 12:24 am

  21. RT @fredposner: @KarlBode did you have any take on the Amazon EC2 SIP attacks and Amazon's (lack of) response? http://bit.ly/ec2sipattack

    mjgraves

    17 Apr 10 at 12:47 am

  22. [...] Following is an edited summary from VoipTechChat.com [...]

    Amazon EC2 Used for Flood Attacks from the Cloud | VoIP Users Conference

    17 Apr 10 at 4:55 am

  23. [...] been reported to Amazon, which has been surprisingly quiet about the matter. The issue has been well documented by one of the attack victims on his blog. The matter was also discussed on the April 16th issue of the VoIP Users Conference [...]

    SIP Attacks From Amazon EC2 Going Unaddressed | JetLib News

    17 Apr 10 at 9:01 pm

  24. [...] his VoIP Tech Chat blog Fred has documented with outstanding clarity his attempts to report the attack that he has suffered [...]

    Graves On SOHO VoIP » Amazon: You Got Some ‘Splaining To Do

    17 Apr 10 at 10:28 pm

  25. Your experience with Amazon sounds a bit like mine, by the way. I work for a regional ISP, provider of IP transit services, our own intranet IP PBX, and several customers’, were attacked by some EC2 IPs including 204.236.221.155, we detected that this attack started the evening of April 9th Eastern time. Massive number of SIP udp packets, SIP registration requests — attempting to gain access by brute force, we presume.

    Our hint of the attack was that the firewall MRTG graphs showed over 60,000 open UDP connections from that IP.

    We blocked the IP at firewall, and contacted the Amazon abuse addresses listed as the abuse contact from WHOIS’ing the IP via ARIN. Nothing became of our report, that we could see, there was no useful response from a human.

    We watched the Firewall logging shunned packets from that same IP for approximately 7 days started late in the evening on April 9th, and on April 16th, we got fed up with it, and blocked the entire /24 at the upstream.

    But it is interesting that the brute force attempts to register with SIP were continuing at a modest rate (a barely noticeable 300 packets per second or so), even after we sent them to /dev/null.

    During that entire 7 day time period, during the continuous ongoing packet flood, there was no useful response whatsoever from a human at Amazon, and they obviously didn’t do anything about the abuse, nor did e-mailing other Amazon addresses seem to help.

    I feel they should have no difficulty whatsoever identifying who was doing this stuff, as long as they keep proper logs of IP assignments and do their frickin due dilligence!

    They are the ISP, so they should act like an ISP.
    I don’t want to be put “in touch with their customer”, ever, that is an unacceptable response.

    As an ISP they should be investigating the abuse at a network level and turning off anything that needs to be turned off to stop immediate abuse, and warn/inform the customer.

    I am only lead to conclude they don’t really mind hosting script kiddies or care to be all that diligent about preventing evil use of EC2. :-(

    Dracolith

    17 Apr 10 at 10:32 pm

  26. I recall reporting a fraudulent credit card transaction that originated from an EC2 address to Amazon in 2008 and they dealt with it very quickly. Since then my experience has been that Amazon’s EC2 support has gone dramatically downhill.

    An engineer in a VoIP service provider I’m involved with had the same SIP brute force attack and pretty much exactly the same experience trying to get Amazon to deal with it (although he was nowhere near as persistent as you and in the end just blocked the subnets).

    On the other side of the coin I actually host a public SIP service on EC2 and I was already in the process of migrating the service due to ongoing technical issues, having to get the service unblocked on various SIP providers was rubbing salt in the wound. I too will be saying goodbye to Amazon in the near future.

    It’s sad to see the pioneers of the cloud descend into ineptness.

    Aaron

    Aaron Clauson

    18 Apr 10 at 3:57 am

  27. For starters, Amazon asked you for formatted logs at the relevant points not just an attachment of the logs.

    Secondly, this attack could happen from anywhere, and PBXs have been getting hammered from China and a wide variety of other locations. It’s not an EC2 phenomenon.

    Thirdly, I simply cannot fathom why you want to get an ‘interview’ from someone at Amazon unless you simply wanted to make this out to be a bigger issue than it really is. She’s right, there is nothing more that she can tell you.

    Segedunum

    18 Apr 10 at 7:10 am

  28. It would be interesting to duplicate this with another cloud provider such as rackspacecloud (just pulling the first name I thought of, never used them).

    Adrian Hensler

    18 Apr 10 at 7:25 am

  29. RT @AmazonEC2: EC2-repost Amazon EC2 SIP Brute Force Attacks on Rise, http://bit.ly/cVvEiz: Amazon EC2 SIP Brute Force Attacks on Rise, …

    VoIP Randulo Zeeek

    18 Apr 10 at 8:00 am

  30. @Segedunum

    > For starters, Amazon asked you for formatted logs at the relevant points not just an attachment of the logs.

    I only provided them a short sample, 40 lines of text filtered to relating specifically to this attack. It would be unacceptable to send an entire log and expect a noc to filter through it without being specifically asked to do so.

    > Secondly, this attack could happen from anywhere, and PBXs have been getting hammered from China and a wide variety of other locations. It’s not an EC2 phenomenon.

    Never said it was an EC2 phenomenon. That being said, Amazon is not china. A complaint to amazon about an attack coming from their network should be handled better.

    > Thirdly, I simply cannot fathom why you want to get an ‘interview’ from someone at Amazon unless you simply wanted to make this out to be a bigger issue than it really is. She’s right, there is nothing more that she can tell you.

    I appreciate your opinion. I had asked specific questions which went unanswered. Additionally, the attack had not stopped. Lastly, their complaint form was broken. She could have helped by stopping the attack, fixing the form, and answering our questions.

    Fred

    18 Apr 10 at 12:15 pm

  31. [...] this issue visible by voting it up on SlashDot. If you haven’t followed out discussions, see Fred’s story. Asterisk user mailing list has a lot of info on it as well. Post on Twitter, their robot stupidly [...]

    Criminal SIP Attacks from Amazon EC2 | VoIP Users Conference

    18 Apr 10 at 1:20 pm

  32. [...] I wrote on VoIP Tech Chat about a SIP Attack one of my servers received from Amazon’s Cloud service. First, Don’t Panic. [...]

    Why I am Boycotting Amazon.com | Fred Posner dot com

    18 Apr 10 at 3:09 pm

  33. SIP attacks meet the Cloud. http://shortn.me/zv4

    Warren Adelman

    18 Apr 10 at 4:17 pm

  34. Fred, you are rightfully upset about the attacks on your system, but you are misguided in your attempt to leverage that into a broader interview with Amazon over whatever issues they may or may not be having with other accounts. If you merely wanted to know more about the steps being taken to resolve your own case, that would be understandable. But, you are crazy if you think that any well advised company will just hand over the details of other accounts, especially in an open interview to a pissed off customer. You unmask your intentions up with your pronoun: “our” questions, Fred? And I don’t want to hear about you acting on behalf of so-and-so unless you have their signature on that letter you sent Amazon asking for an interview.

    Matt

    19 Apr 10 at 10:24 am

  35. @Matt…

    “Our” isn’t a cover up. It’s for myself & Patrick… who was also on the line when we called and left messages. Like the tagline says… “Patrick and Fred Chat… sometimes about VoIP.”

    I may be crazy, but I never asked for details on other accounts… although from her statement, they would normally connect me to their customer to see if the traffic was legitimate. My request for an interview was to get those questions I sent her in an email answered. They have not been.

    Fred

    19 Apr 10 at 10:41 am

  36. RT @geekandi: SIP brute force attacks from Amazon EC2: http://bit.ly/ahXjNH

    Jazz blues wine voip

    19 Apr 10 at 5:40 pm

  37. Amazon posted something on their website about this incident and how they have learned from this:
    https://aws.amazon.com/security/

    Frank

    Frank

    20 Apr 10 at 5:23 pm

  38. [...] Amazon EC2 SIP Brute Force Attacks on Rise. [...]

    Network Security Podcast » Blog Archive » Network Security Podcast, Episode 194

    20 Apr 10 at 7:53 pm

  39. [...] Amazon EC2 SIP Brute Force Attacks on Rise. [...]

    Network Security Blog » Network Security Podcast, Episode 194

    20 Apr 10 at 7:55 pm

  40. [...] The documentation from Princeton on this issue amazes me. It is extremely detailed and may just have raised the bar for incident reports. They have published steps to reproduce the issue as well as a workaround. AND… they made these documents available (easily) to the public. There’s no chest beating, hyperbole, or exaggeration. Just a detailed “this is the problem, this is what we’ve done” document. Amazon should really follow their example. [...]

    Princeton: No Love for iPad (But no Ban either) | VoIP Tech Chat

    21 Apr 10 at 7:48 am

  41. Amazon EC2 Attacks, and smoke screen continues http://lnkd.in/hDa6QC

    Jazz blues wine voip

    21 Apr 10 at 5:03 pm

  42. [...] Amazon EC2 SIP Brute Force Attacks on Rise [...]

    Links 15/4/2010: Linux Foundation’s Collaboration Summit, Dragora Linux 2.0, ZEN-mini 2010 | Techrights

    22 Apr 10 at 3:53 pm

  43. [...] Amazon EC2 attacks: just a PR damage control post or two, no real action. [...]

    Amazon EC2 #fails to Investigate Attacks | VoIP Users Conference

    24 Apr 10 at 3:07 am

  44. [...] This post was mentioned on Twitter by WW Ben Franklin Do?. WW Ben Franklin Do? said: RT @fredposner: Amazon EC2 SIP Brute Force Attacks on Rise http://bit.ly/bWw1VQ [...]

    Tweets that mention Amazon EC2 SIP Brute Force Attacks on Rise -- Topsy.com

    28 Apr 10 at 8:18 pm

  45. Fred you have done a good job on trying to track down answers. It really surprises me when people pull stuff out of no where to try and derail your progress.

    You never said anything about asking about accounts, that was just yanked out of the depths of.. well you know Matt…

    Also, Segedunum sounds like some Amazon employee trying to protect their interest. I’m not saying directly he/she is but come on read your own comments, nearly everything you said was pulled out of no where and not even one thing was accurate.

    If you are going to attack what Fred is doing, at least get the facts right. Otherwise you look like a douchebag.

    Keep up the good work Fred.

    Avery

    13 May 10 at 9:59 pm

  46. Fred -

    Well as of this morning, I can confirm that attacks from Amazon continue. We saw a compromise of an extension that originated from EC-2 (79.125.71.218). The successful authentication to the system was at 10:58am EST on 5/15/10.

    What is interesting is that this system was not used to make any calls – instead they originated from another IP, based out of Germany. This other IP (91.194.85.241) also appeared to be brute forcing as the client was ‘friendly scanner’, which is sipvicious from what I can see.

    We reported both to the related abuse contacts, but like other submissions, I don’t expect a lot of response.

    I hope that Amazon is actively working on these – I recognize the challenge they have given the number of systems they manage. But active cooperation and protactive monitoring for abnormal traffic seem to be an obvious necessity if one is to run a system like they do.

    Rick

    15 May 10 at 3:04 pm

  47. [...] over a month ago, we reported that SIP attacks from the Amazon EC2 cloud were on the rise. While the attacks we received last month were limited to “extension only” registration [...]

    SIP Attacks From Amazon EC2 Cloud Continue | VoIP Tech Chat

    16 May 10 at 5:11 pm

  48. This is a bit of concern about Amazon cloud services (by @fredposner): http://bit.ly/bhyW5x

    Joe Devon

    19 Jun 10 at 6:37 pm

  49. I’m currently being deluged by SIP register packets from an host apparently hosted by gogrid.com, so no doubt the cloud providers are going to repeat all the mistakes of the ISPs and email providers in hosting and facilitating malicious customers.

    VK

    23 Aug 10 at 5:23 pm

Leave a Reply