Attacks from the cloud.

Just over a month ago, we reported that SIP attacks from the Amazon EC2 cloud were on the rise. While the attacks we received last month were limited to “extension only” registration attempts, one of the attacks we received this morning included what we assume was a standard dictionary attack.

The first attack came from 204.236.245.101. In less than 60 seconds, this IP attempted more than 11,500 registrations against our server. Most of these were 4 digit extensions (download the log (zipped) here). The second attack came from 184.73.4.183. In less than 90 seconds, this IP attempted more than 21,000 registrations against our server; including what we think is a standard dictionary attack complete with root, postmaster, pixadmin, etc. (download the log (zipped) here).

From past experience, Amazon will simply not do anything about these attacks. The only way to contact Amazon to report abuse is through their web form, which results with Amazon either completely ignoring you or sending a delayed response asking for the exact information you have already sent (and then ignoring you).

So, I’m looking for more ideas from the community on how we can get Amazon to help us stop their network from leveraging a very powerful attack against little ol’ SIP servers.

We are currently deploying a custom perl script to block these attackers via iptables (which is why the attacks registration attempts “stopped”).

Thanks for reading and we’ll be updating this as soon as more information comes in!

Update #1

The first response from Amazon:

Thank you for submitting your abuse report.

We have completed an initial investigation of the issue and learned that the IP address you reported did indeed belong an Amazon EC2 instance. These intrusion attempts that you report were not, however, initiated by Amazon.

One of the biggest advantages of Amazon EC2 is that developers are given complete control of their instances. While the IPs may indicate that the network is Amazon’s, our developer customers are the ones controlling the instances. You may learn more about EC2 at http://aws.amazon.com/ec2

That said, we do take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use. We’ve already contacted the Amazon EC2 customer who controlled the instance in question and informed them that they are required to terminate their unauthorized interaction with your network, failing which we will terminate their instance. In cases of egregious abuse or as we otherwise deem appropriate, we will immediately terminate all their instances and suspend their account.

If you have blocked this address range, please be aware that usage on the address range is transient and new users may soon be operating from those addresses and may not be able to reach you; once you have confirmed that the activity has been ceased by our customer, you should open your filters to re-allow traffic.

Thanks again for alerting us to this issue.

Original report:

* Source IPs: 204.236.245.101
* Abuse Time: Sun May 16 08:53:00 UTC 2010
* NTP: Y

How can I send a message to the EC2 customer?
Complete and submit the web form here.

How can I contact a member of the Amazon EC2 abuse team?
Send an e-mail to ec2-abuse@amazon.com to contact a member of the Amazon EC2 abuse team.

Please note: This e-mail message was sent from a notification-only address that cannot accept incoming e-mail. Please do not reply to this message.

Amazon Web Services

If you feel you are receiving this email in error and do not wish to receive further notifications, send an e-mail to ec2-abuse@amazon.com.

Amazon Web Services LLC is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message produced and distributed by Amazon Web Services, LLC, 1200 12th Ave South, Seattle, WA 98144.

I really don’t need a sales pitch in my abuse response.

Additional Information:

• Automatically Block Failed SIP Peer Registrations (Team Forrest)
• Amazon EC2 SIP Brute Force Attacks on Rise (VoIP Tech Chat)
• SIP Brute Force Attack Originating From Amazon EC2 Hosts (Building The Net)
• Properly stopping a SIP flood (joshua stein)

Attacks from the cloud.

Update #1: 12 APR 2010. Amazon NOC’s response.
Update #2: 12 APR 2010. Amazon Statement.
Update #3: 13 APR 2010. Amazon Response.

Complaints of rampant SIP Brute Force Attacks coming from servers with Amazon EC2 IP Addresses cause many admins to simply drop all Amazon EC2 traffic. Generally, SIP brute force attacks attempt to register various peer names to a system and/or attempt to guess passwords of known/guesses peers or endpoints.

The complaints mentioned this weekend show an excessive amount of traffic; with some providers claiming 6GB of traffic dedicated to such attacks. Since we ourselves received an attack from an Amazon hosted server, we also reported and complained to the Amazon NOC/Abuse depts. As of this posting, no response or acknowledgement has been received from Amazon. The response from Amazon is below.

There are various techniques to assist with minimizing DDoS and Brute Force attacks, such as limiting access via the public internet, using strong passwords, not mapping extension name to peer/endpoint name, limiting simultaneous calls, and aggressively monitoring usage. Automatic blocking of abusive IP’s (fail2ban, blockhosts, etc.) can also assist with minimizing damage.

Update #1: 12 APR 2010. “Response” from Amazon’s NOC

So when this happened, I submitted a report to Amazon complaining of the attack. The report was sent to their abuse and noc mails and contained the standard abuse report, including their host, my host, the protocol, ports, and description of activity; as well as a sample log.

About 48 hours later, they sent this as a response:

From: Amazon.com <ec2-abuse@amazon.com>
Subject: Your Amazon EC2 Inquiry
Date: April 12, 2010 7:31:59 AM EDT
To: Fred Posner

Hello.

Thank you for contacting Amazon Web Services. We take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use.

Because Amazon EC2 Public IP addresses may change ownership frequently, without additional information we will be unable to identify the correct owner of the IP address for the period of time in question.

So that we can process your report and identify the actual customer in question, we require the following information. Please note that we will not open attachments under any circumstance

• Source IP
• Destination IP (your IP)
• Destination Port and Protocol
• Accurate Date, Time and *Time Zone* of activity
• Intensity and frequency of activity in short log extracts, no larger than 4KB
• Your contact details (phone and email)

For a faster response, please file your report using the AWS Abuse form at the link below:

https://www.amazon.com/gp/html-forms-controller/AWSAbuse/

We appreciate your help in providing the necessary information requested.

Best regards,

-EC2 Abuse Team

So, their laziness aside (since the report to them included ALL the information requested), I filled out the form– which of course failed with an unknown error. (below)

For people interested in moving to Amazon’s cloud, this is a good example of the quality of people conducting your network administration. I’ve sent another response by email. Others have blocked the entire EC2 IP range and have requested their upstream providers do the same.

So my question to you… What would your next step be?

Update #2: 12 APR 2010. Amazon Statement.

Following a request for interview/statement, VoIP Tech Chat received the following communication from Amazon’s Public Relations Manager:

Hello Fred and thank you for contacting us.  Over the weekend, we received a report of a suspicious account and began an investigation.  Our normal process is to connect the two involved parties to give them an opportunity to talk in case the abuse is not malicious but is simply heavy traffic from a legitimate customer.  If that is not successful, we then move to isolate the traffic from the abusing party.  Normally this process works quite well for situations our customers have encountered, however this incident has highlighted the need for an escalation process to address potentially malicious attacks more quickly.  Additionally, we are working on quickly putting better protections and processes in place to better guard against unwanted SIP traffic.  We take the security of our customers and our quality of service very seriously, and will  continue to work to improve our processes and services for customers.

Thanks

Kay Kinton
Public Relations Manager
Amazon Web Services

I’ve replied to again ask for an interview and will update if a response is received. The statement states this was over a weekend, however doesn’t address that the attacks continued today. It also states a “report” was received, but there were many reports submitted. That being said, at least they responded.

Update #3: 13 APR 2010. Amazon Response. Decline of Interview.

After Kay Kinton’s statement, I asked her for a phone interview.

Sent: Monday, April 12, 2010 2:00 PM
To: Kinton, Kay
Subject: Re: Amazon Web Services

Kay,

Thank you for your statement. I would like to interview you about this for VoIP Tech Chat… it would be an over-the-phone interview and would be for 5 minutes or however much longer you would like.

—fred

Kay’s response was quick, and to the point:

What else can I tell you Fred?

I truly dislike email for interviews for no other reason that not getting the tone of that response. Did Kay mean that as “Sure, great! What else can I help you with?” Or, was it more along the lines of, “I answered you. What now?”

Giving her the benefit of the doubt, I replied:

Date: April 12, 2010 5:24:14 PM EDT
To: “Kinton, Kay”
Subject: Re: Amazon Web Services

We would like to interview on this. I thank you for the statement, however I have additional questions:

I know of 12 complaints since Saturday (from different reporters) that were submitted regarding SIP attacks from EC2 to outside systems. How many complaints did you receive since Saturday?

I know attacks continued today and may even be ongoing. There were attacks as of 1pm EST hitting systems with over 640K of data. Are you still seeing attacks? How many hosts were identified?

Were the attacks submitted from one customer/client of yours or many?

Those are my initial questions, however I do request a phone interview rather than email. I find them much easier to exchange information as well as generally a better expressive forum for an interview.

—fred

Good thing I didn’t hold my breath. The next day, after not receiving a response, I called Kay several times and emailed her for an update. Her response via email:

Hello Fred. We believe that we’ve identified and shut down the illegal activity and are closing the loop with customers. We’d certainly be interested in hearing of the cases you refer to below so we can follow up.

I tried reaching out to her but have not had responses. Which leaves me with this…

Her response did not answer my question and I certainly have no basis to believe that Amazon is currently taking any interest in this matter. They’ve told us prior that they cannot pinpoint IP to timeframe as well as that during an attack, they’d try to mediate between parties rather than actually stopping the attack in progress (to give them an opportunity to talk). Sadly… when I’m being flooded, I want the flood to stop. Afterwards, I’ll be glad to talk. But I digress…

Since Kay did not answer any of the additional questions we asked, but did state that she’d be interested in hearing about the other cases, we will encourage anyone with information or feelings about this issue to contact Kay Kinton directly:

Kay Kinton
kinton@amazon.com
Public Relations Manager
Amazon Web Services
Phone: 206-266-8387

For More Information:

• Asterisk User’s Mailing List Archives
• 7 Steps to Better SIP Security (Digium)
• Blockhosts
• Voice over IP Security Alliance (VOIPSA)
• SIP Brute Force Attack Originating From Amazon EC2 Hosts (Building The Net)
• Properly stopping a SIP flood (joshua stein)
• Automatically Block Failed SIP Peer Registrations (Team Forrest)

Report your error with our form. Fail.