Fanatical Support

Got Rackspace? Got WordPress? If so… you may just have a problem.

We’ve been getting calls today from Rackspace clients (hosting WordPress sites) that have been compromised similarly to the GoDaddy hack a few weeks back. The Unmask Parasites Blog has an excellent article on the attack posted on their, well, their blog.

There are some huge sites that have been hit, and some not-so-large as well (we personally were hit by an earlier attack). In the “Is Cloud the answer” debates, this will surely become an example of how a compromise in the cloud, can devastate an entire farm.

Update 6/19/2010

Shortly after this article was initially posted, Rackspace via their Rackcloud Twitter account posted the following message:

Reports, schmeorts.

Of course… details never came. I tweeted them myself (Oh no you didn’t… Oh yes I did):

Show me the money.

At this point, they haven’t replied to my request or posted any additional information on their twitter account. I think they moved on… the next day they were more interested in talking about how “Cassandra by Example translated to Japanese!”

Also, one day… one day I’ll spell check my tweets. Until then, read at your own grammatical risk.

Read more:

http://blog.unmaskparasites.com/2010/06/14/attack-on-wordpress-blogs-on-rackspace/

Last week, Ward Mundy, Tim Panton, Karl Fife, Leif Madsen, Yours Truly, and many other regulars discussed a SIP Caller ID Injection Hack. As in all conversations, opinions differ. My position about where to best filter this injection differed than Ward Mundy’s thoughts… and, courtesy of the VoIP Users Conference, you can listen to the conversation and form your own opinions.

Although, next time… maybe you’d enjoy actively participating in our conversations rather than listening to the replay.

→ SIP Hacks: who should filter what, where? (VoIP Users Conference)

(The VoIP Users Conference provides weekly live discussion about VoIP, SIP, Asterisk and all kinds of telephony-related topics every Friday at 12pm EST. For more information, please visit http://vuc.me.)

Whoops. My bad.

LifeLock promises to “take control” of your identity — they just don’t tell you who gets to take control. Patrick and I chatted a while back about Todd Davis, the CEO of LifeLock, and how his ads promoting the ability of his company to protect identity, actually helped with the theft of his own. Back in 2007, a gentleman in Texas had used Davis’ identity to obtain a $500.00 without Davis’ knowledge. In fact, Davis only had learned about it after the unpaid loan was sold to a debt collection agency — but that’s old news.

Today, thanks to the Phoenix News Times, we learn that Davis had his identity stolen a grand total of 13 times. Or, at least 13 times that we know of.

With attention grabbing ads that published Davis’ Social Security Number, LifeLock caught the attention of many customers; as well as the FTC — who accused the company of running a scam operation and fined them $12 million dollars.

Additional Reading

• Cracking LifeLock: Even After a $12 Million Penalty for Deceptive Advertising, the Tempe Company Can’t Be Honest About Its Identity-Theft-Protection Service (Phoenix New Times)
• Lifelock Dinged $12 Million for Deceptive Business Practices (wired.com)

Attacks from the cloud.

Just over a month ago, we reported that SIP attacks from the Amazon EC2 cloud were on the rise. While the attacks we received last month were limited to “extension only” registration attempts, one of the attacks we received this morning included what we assume was a standard dictionary attack.

The first attack came from 204.236.245.101. In less than 60 seconds, this IP attempted more than 11,500 registrations against our server. Most of these were 4 digit extensions (download the log (zipped) here). The second attack came from 184.73.4.183. In less than 90 seconds, this IP attempted more than 21,000 registrations against our server; including what we think is a standard dictionary attack complete with root, postmaster, pixadmin, etc. (download the log (zipped) here).

From past experience, Amazon will simply not do anything about these attacks. The only way to contact Amazon to report abuse is through their web form, which results with Amazon either completely ignoring you or sending a delayed response asking for the exact information you have already sent (and then ignoring you).

So, I’m looking for more ideas from the community on how we can get Amazon to help us stop their network from leveraging a very powerful attack against little ol’ SIP servers.

We are currently deploying a custom perl script to block these attackers via iptables (which is why the attacks registration attempts “stopped”).

Thanks for reading and we’ll be updating this as soon as more information comes in!

Update #1

The first response from Amazon:

Thank you for submitting your abuse report.

We have completed an initial investigation of the issue and learned that the IP address you reported did indeed belong an Amazon EC2 instance. These intrusion attempts that you report were not, however, initiated by Amazon.

One of the biggest advantages of Amazon EC2 is that developers are given complete control of their instances. While the IPs may indicate that the network is Amazon’s, our developer customers are the ones controlling the instances. You may learn more about EC2 at http://aws.amazon.com/ec2

That said, we do take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use. We’ve already contacted the Amazon EC2 customer who controlled the instance in question and informed them that they are required to terminate their unauthorized interaction with your network, failing which we will terminate their instance. In cases of egregious abuse or as we otherwise deem appropriate, we will immediately terminate all their instances and suspend their account.

If you have blocked this address range, please be aware that usage on the address range is transient and new users may soon be operating from those addresses and may not be able to reach you; once you have confirmed that the activity has been ceased by our customer, you should open your filters to re-allow traffic.

Thanks again for alerting us to this issue.

Original report:

* Source IPs: 204.236.245.101
* Abuse Time: Sun May 16 08:53:00 UTC 2010
* NTP: Y

How can I send a message to the EC2 customer?
Complete and submit the web form here.

How can I contact a member of the Amazon EC2 abuse team?
Send an e-mail to ec2-abuse@amazon.com to contact a member of the Amazon EC2 abuse team.

Please note: This e-mail message was sent from a notification-only address that cannot accept incoming e-mail. Please do not reply to this message.

Amazon Web Services

If you feel you are receiving this email in error and do not wish to receive further notifications, send an e-mail to ec2-abuse@amazon.com.

Amazon Web Services LLC is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message produced and distributed by Amazon Web Services, LLC, 1200 12th Ave South, Seattle, WA 98144.

I really don’t need a sales pitch in my abuse response.

Additional Information:

• Automatically Block Failed SIP Peer Registrations (Team Forrest)
• Amazon EC2 SIP Brute Force Attacks on Rise (VoIP Tech Chat)
• SIP Brute Force Attack Originating From Amazon EC2 Hosts (Building The Net)
• Properly stopping a SIP flood (joshua stein)

It’s a GREAT read and I leave you with this: Magic Button. (read the article)

Related links:

• e4 Technologies
• Michael Graves on SOHO VoIP

†Yes… Wicked cool and useful. If it doesn’t meet the criteria for both, it doesn’t make his blog.

Our Hero

Early humans found hollowed out rocks to turn into homes, originating the term “Cave men”. ¹ This constraint made community difficult, so humans advanced to creating homes from natural materials, such as wood. Primitive homes were modeled on the cave, with nothing but some closed walls and an uncovered opening. Thousands of years of evolution lead us to create doors that open, close, and lock, and windows that allow us to see out and in, then glass to keep what’s out out and what’s in in, then curtains to cover what’s both out and in. In the end, we have the same caves we had before, with our darkness and privacy.

In the 1600′s the Dutch East India company was like the Wal-Mart of the high seas. If you worked on a ship for the DEI, actually called VOC, but let’s not have an acromania tournament over it, you lived day in and day out with the other people on the ship. Everyone knew everyone’s business, and that’s just how it was. There would be no need to do a status update when you went to the head, because everyone watched you go.

With the onset of industrialization and assembly-style production in the 1900′s, factories became central to small towns and people began working together, but their was a similar environment of everyone knew everyone’s family and friends and kids and lifestyle. There just weren’t a lot of secrets. Only in the last 50 years have we moved to the cubiclised, white-collar, technically-oriented jobs where turnover is an expectation and no one really bothers to get to know everyone else. Cliques form, but on the whole there isn’t a sense of community.

In a relatively short span of time, we created a generation and a culture that has a “right to privacy.” We have seen this concept denied by courts who say employers can regulate lifestyle as a condition of employment, and that what an employee does outside of work can still be used against her at work. Drinking, drugs, cigarettes, and even functions allowed to be attended can all be used as conditions of employment in our “right to work” world.  Though it has been upheld time and time again, the belief in this right grows ever stronger.

The political buzzword of the last decade has been “transparency.” We the people should have an open window on the workings of our government, of our corporations, of our financial institutions. We should see how the cogs turn and the deals are made, we should have open access to it all. At the same time, a subculture of companies has grown around controlling the online image of individuals. Ex-boyfriend posted some risque pictures of you? They can fix that. You got fired from your old job for coming to work drunk, and some people decided to blog about it? They can fix that. From the benign to the outright slanderous, companies that specialize in online identity rehab are doing bang up business curing the internet of individuals’ indiscretions.

Should it matter? Should you want to work for a company that would use your facebook status update about hating filing against you in an interview? Does that tweet about being drunk at the Alice In Chains concert make you a bad person or in any way impact your job performance? Are companies better off pretending that their employees don’t have a personal life? Maybe this is the wake up call that companies need to start treating their employees like people. Maybe it’s time to open up the door to the cave and not worry about what others will see, because their cave door is wide open too.

^{1. This is rather vague and unresearched proposition, because this is a tech blog and not an anthropology blog. Please do not blame us when you crib this and fail your class.}

If you want to be judgmental of our thoughts, feel free to follow Fred and Patrick on twitter!

I do love their logo.

Ward Mundy, of Nerd Vittles / PBX in a Flash fame, warns of a FreePBX Security Vulnerability allowing a system to be compromised simply by displaying a CDR report in the FreePBX browser.

There is a very serious security vulnerability that needs to be patched by loading the very latest version of FreePBX Framework as soon as it becomes available for your version of FreePBX. Just displaying a CDR report in the FreePBX browser could compromise your system.

The 2.5 and 2.6 patches already have been released and probably 2.7 as well. Load this patch IMMEDIATELY!!!

Setup, Module Admin, Check for Updates on Line, Upgrade All

2.5.2.3: #4223 Security Vulnerability
2.6.0.2: #3805, #3707, #4188, #4223 Security Vulnerability

For more information, check out the PBX in a Flash Forum.

I R Eatz U R Dataz!

I love my netbook. I love my netbook so much, I have two of them (okay, one is the wife’s). Surprisingly, I managed to survive months on nothing but my netbook doing fairly intensive SQL / VoIP / Web work. The hard drive is a little slow, but the overall performance is outstanding.

When I travel, I can use Skype to video chat with the built in webcam and get great quality (both ways) for both picture and sound. It’s like a giant smart phone. It reminds me of the $1000+ “video phones” that were supposed to be the future of talking on the phone… then people realized they really didn’t want to “get pretty” to use the phone. Now, for around $250 a unit, you can have that and so much more.

Recently, my wife’s netbook gave an error on boot that had something to do with a missing windows file (system32\ntoskrnl.exe). Apparently, she’s not the only one with this issue… it seems to be quite common. Of course, as the techie/geek/nerd of the castle, it was my job to slay this dragon and I came across the only problem I have had with this magical mini machine. Not so much a problem… more like open questions to Acer.

1. To access the hard drive, you have to remove 17 screws (maybe more, I think I lost some extras), you have to remove the keyboard, you have to remove the top, you have to remove this little card on top of the mother board, this side circuit board, the motherboard. Then you slide the hard drive out, reverse. Why not put a panel on the bottom to access the hard drive directly?
2. Why can I find nothing in your documentation about Alt-F10? This is a handy mode that lets you recover the operating system to factory defaults, but I don’t see it in my Acer manual.
3. In line with question 2, Why is the only recovery option to completely reset to factory? Since this is basically a stripped version of the OS, why not offer an explorer window so I can copy files to an SD or USB drive before formatting, or just backup the user files to another partition?
4. What degree in sadism lead to the design of the three little clips that hold the keyboard in place?

All told, from error to recovered could have been done in < 30 minutes, and I HIGHLY recommend Acer products to everyone I know. From their higher-end Ferrari laptop to the humble netbook (go with XP home, Windows 7 netbook edition is crippleware garbage), I have never had a problem with their hardware or software that made me lose respect for the brand, which is saying a lot.

For More Information:

• Acer: Acer Aspire One Website
• Skype: Free PC to PC Video Calls

Whoops. Our Bad.

McAfee released a “faulty update” this morning causing the security program to believe a good file had gone bad. In what the company calls a “False Positive Issue,” the anti-virus software identifies a good windows file, svchost.exe, as the W32/Wecorl.a virus; causing the system to continuously reboot and lose network access.

At the University Hospital in Syracuse, NY 2,500 computers were affected; however the hospital stated that patient care was not compromised. Other public service/safety organizations were also impacted, including the Kentucky State Police, the National Science Foundation, and Illinois State University.

The impact forced several Rhode Island hospitals to stop treatment of non-trauma patients in emergency rooms as well as postpone non-essential surgeries.

McAfee’s Barry McPherson posted on their security blog:

McAfee is aware that a number of customers have incurred a false positive error due to this release. Corporations who kept a feature called “Scan Processes on Enable” in McAfee VirusScan Enterprise disabled, as it is by default, were not affected.

Our initial investigation indicates that the error can result in moderate to significant issues on systems running Windows XP Service Pack 3.

The faulty update was removed from all McAfee download servers within hours, preventing any further impact on customers. We believe that this incident has impacted less than one half of one percent of our enterprise accounts globally and a fraction of that within the consumer base.

For More Information:

• McAfee: McAfee Response To Current False Positive Issue
• ZDNet: Defective McAfee update causes worldwide meltdown of XP PCs
• Syracuse.com: University Hospital computers plagued by anti-virus glitch

Describing what they feel is a bug with the iPad’s operating system, Princeton recently announced (via their Knowledge Base):

Network monitoring has shown that many iPad devices have caused a problem on the campus network. These devices continue to use an IP address they have been leased well beyond the time they should. (In technical terms, the device’s DHCP client software stops renewing its lease, but the device keeps using the IP address after the DHCP lease expires. This is not a WiFi issue.) This behavior causes a disruption on the campus network.

Additionally, Princeton posted an extremely thorough description of the problem. Although many articles and news reports have stated that Princeton has “banned” the iPad from it’s network, Princeton assures the public that this simply is not the case. Initially, only the devices having the issue were banned from the network, and those have been allowed to reconnect via a workaround.

Beginning April 4 2010 (prior to having any workaround), when an individual iPad malfunctioned, we would contact the owner to advise him or her of the problem. When the same iPad malfunctioned a second time, we would block that device from using our network, and contact the owner again; at that time we would advise the owner that the blocks would remain in place until there was a fix from Apple, or there was a workaround to the problem.

On a personal note…

The documentation from Princeton on this issue amazes me. It is extremely detailed and may just have raised the bar for incident reports. They have published steps to reproduce the issue as well as a workaround. AND… they made these documents available (easily) to the public. There’s no chest beating, hyperbole, or exaggeration. Just a detailed “this is the problem, this is what we’ve done” document. Amazon should really follow their example.

Links of Interest

• NJ.com: Princeton Experiences iPad Problems
• Princeton: iPad/iPhone OS 3.2 Stops Renewing DHCP Lease, Keeps Using IP Address
• Princeton: Workaround for “iPad/iPhone OS 3.2 Stops Renewing DHCP Lease, Keeps Using IP Address” Issue