December 2010 Hacking
       By now, many of you know that Fur Affinity was attacked on Thursday, December 16th 2010. Attackers were able to compromise the admin system using an XSS exploit in the trouble ticketing system to gain control of an admin account. We pulled the website offline, and closed the hole that lead to the initial attack, but not before the intruder was able to illegally compromise the private notes of 41 users (including admins and staff) and the vandalism of several galleries. Regrettably, the leeching of notes occurred before the hacker made his presence known on the site, and we were not able to stop it.

       At no point were user passwords or the site database compromised.

       After closing the initial hole that the intruder was using to compromise the site, they then attacked an admin's e-mail, managing to compromise their email account to perform a password reset. With the new password, they were able to get back into the site and into the forums. At the same time, another attack was launched on a second admin, compromising a long-abandoned account they had which was setup as an e-mail fallback for their main account. In both instances, the attackers were able to gain access back into the system, causing scattered vandalism.

       We were able to flush the attacker out of the system through multiple wipes of cookies and active login sessions (which some of you may have noticed when your account were logged out).

       After Yak revised and recoded the security side of the admin panel, the attackers then launched a distributed denial of service (DDOS) attack against FA as a final measure. Working with our host, we were able to block the attack and restore services to the site. While we had initially suspected potential issues due to the 1.2 million Gawker passwords that were leaked (which had affected some regular users on the site), we want to clarify that the Gawker leaks WERE NOT an issue with the FA intrusion.

       Galleries which were wiped are in the process of being restored, and we are working to strengthen and improve security. We have also removed the ability for certain admins to view notes. We will be bringing in additional coding help to perform security audits and improve upon the site's platform, as we do take security seriously. We regret that this happened, and ultimately the blame for this lay with us for letting the hole slip through the cracks. That said, it does not excuse the intruders for their actions, and we are working with law enforcement to pursue the issue.

       On behalf of the entire staff of FA we apologize for what happened. We make no excuses for what happened.

----------------------------

       If you have questions, please feel free to ask, we will update the thread with a Q/A. Keep responses civil, and honest. There has been enough drama over this, and we want to work towards peaceful resolution.

EDIT: Had the date wrong in the initial attacks. My apologies. The initial incident happened on Thursday, not Friday.

(1655 Comments |Comment on this)

Site back online.
You probably knew that already, though.

(35 Comments |Comment on this)

Forums are back online.
Thanks to Carenath donating space on his server, the forums are back online.

Now, I will preemptively answer some questions.

• This wasn't done earlier because I didn't have the data off the old forum server until late, late Sunday night.
• No, we can't use Carenath's server to bring the mainsite up. The main site is about a thousand times larger than the forums. (Both in terms of disk space and bandwidth requirements.)
• We have everything we need to install the servers at the new facility. The only concern now is weather related, but that shouldn't be a problem. (Incidentally, did you know that studded tires are illegal in Maryland? This is highly relevant.)

(92 Comments |Comment on this)

Good News, Bad News, Good News, Bad News
Good News: We're all ready to go on our end. We have the IPs, we've got the rack setup at our new colo. All we need to do is move our servers into the facility, update the IPs and config the DNS. That'll take maybe 30 to 45 minutes.

Bad News: Switch and Data (hey, remember them?) were not able to process our badges in time, so we're not able to get into the colo. Yet. Fuck. You may proceed to queue the "BADGES?! WE DON'T NEED NO STEENKIN' BADGES!" quotes, because... that's the only thing holding us up. THE ONLY THING. Only thing. Yep. That. That thing? That's the only thing. THE. ONLY. THING.

Only thing.

Good News: Net-cat and myself (and maybe Dax) will be meeting at the colo at 5:00PM EST tomorrow to setup, install, configure and bring Fur Affinity online. We had hoped to do this today, but Murphy's Law is a fickle bitch unfortunately the paperwork pushers didn't pushpushpush.

Bad News: I'm out of Cherry Bawls. FFFFFFFFFFFFFFFFFFFFFFFFFFFF

Current Mood: annoyed

(267 Comments |Comment on this)

FA Status Update
Just a note:
Net-Cat drove over to my place tonight and we did pre-maintenance on FA's servers (Tiamat, Bahamut, Trogdor and Novastorm) and all systems are good to go. We updated some of the software on Novastorm (VMWare) to the latest version.

As of right now, everything is solid on our end. We are currently waiting for the colo facility to process our paperwork and get the billing process started. That's our only hold up right now. :) No, we do not have an ETA at this time, but we're still on track for our original forecasts. Again, everything is good to go on our side.

At this point it's all paperwork.

(102 Comments |Comment on this)

It looks like we're planning to bring the site up Monday night. While we were originally aiming for this weekend, we're currently waiting on the paperwork to process at our new colo facility. While the tech team works 24/7 the back office team is a 9-5 crew. This may turn out to be a good thing as they are expecting snow, and the first snow of the season can usually be kind of hazardous road condition wise. =P

We'll keep you updated as usual.

On the plus side here's the good news: we're upgrading to a 100Mbit dedicated line, which is roughly 25% more bandwidth. In addition, our new deal allows Fur Affinity to operate its main data center at roughly 40% cheaper costs. This will allow us to further invest money into better hardware, upgrades and permit us to do off-site backups.

(Note: With all due respect to Canada, our new colo facility shall remain in Virginia.)

(184 Comments |Comment on this)

Fur Affinity will be returning [this space intentionally left blank]*.

* Update shall be later tomorrow to fill in the gap

---------------------

I was able to down to the root of things. The entire outage goes down to a hardware problem on XO Communications. XO Communications did not have replacement hardware for the fiber lines that went down, and had to order it. Much like Comcast, when they say "we'll have a technician there at 2pm" and by the time 6pm rolls around he hasn't shown up, XO Communications dropped by ball.

The entire issue was supposed to have been fixed by Tuesday night. This entire outage can, should and will be blamed on them. I'm not going to excuse the total lack of communication on our host's part, but that's another story altogether.

I sincerely apologize for the outage, the inconvenience and frustration this has caused. This is not how we want FA to be known, and we've been striving to improve everything, and we'll be doing just that in the coming months.

I have more I could say, and say venomously, but I'm going to hold my tongue. I've been working on FA and pulling servers up until midnight. I'm tired, and shall retire to bed. For now, adieu.

(132 Comments |Comment on this)

Site Status
Just as an FYI - as I posted to the FA Twitter, we have put out a few quotes and bids on alternative hosting for other colos. We find the current situation to be unacceptable, and do not approve of how things are being handled.

While we we do not have new information at this time, we're checking into and examining all available options.

Update 1 - 02DEC@12:01 EST
We've found a solid co-location facility in Montreal, Canada. I am waiting to hear back from some other quotes at this time.

Update 2 - 02DEC@13:55 EST
Talking to two other colos - one of which is in walking distance from my apartment. They local one is offering a deal that may be "too good to be true" due to recent expansion (they're trying to fill space ASAP, so the deal may be legit) but we're investigating it. It *is* from a legit company that's been in business for 10+ years, so... we'll check it out.

More detail as follows. We're not rushing into this, and we want to do things right.

Update 3 - 02DEC@18:41 EST
I have a scheduled tour of a new colo tomorrow. Keep your fingers crossed. If it's as good as everything appears to be be, this will not only be a professional colo/hosting solution (with a promised 99.9% uptime backed by an SLA) but will save Fur Affinity quite a bit of money annually.

We will keep you informed.

NOTE: I know we're even not three days into this outage (Outagegate), but I would like to re-iterate one thing: this will NOT be an issue like last August. We will not need donations, nor ask for them. If anybody tells you otherwise you have our permission to slap them.

(365 Comments |Comment on this)

[Updated 2009-12-02 02:00 UTC] Oh, and you thought your cable outages were annoying?
Imagine how we feel when our ISP, which supposedly has redundant links, seems to have perennial peering issues.

Pretty much the only way you're going to be getting your wholesome, furry goodness *cough* at this point is by throwing a party in the server room. (And I'd imagine that S&D would take issue with that.)

tl;dr - ISP's shit's broke. Nothing we can do about it. Patience is a virtue. Etc, etc, etc.

PS: I totally just noticed that I haven't done stats in like... two months. Not that I can do it now. Servers are inaccessible. :D

Lack of update: 2009-12-01 15:30 UTC (10:30 EST)

Well, what can I say. Our ISP is still not talking to us. (And I'm sure anyone who is resourceful can dig up who our ISP is.)

Going on 24 hours with no updates. When we know something, we'll post it.

We also have no plans to ask for donations for this problem. Because it won't actually solve the problem, in this case.

Update: 2009-12-02 02:00 UTC (21:00 EST)

(08:48:04 PM) Dragoneer: That the fiber lines were damaged, they're doing emergency repairs.

No estimate.

(527 Comments |Comment on this)

More routing issues. 2009-09-14
We've had reports of people outside the US having trouble accessing the site.

This is due to routing issues at an upstream provider.

Cages have been rattled. All we can do now is wait.

Sorry for the inconvenience.

(2 Comments |Comment on this)

Monthly Statistics, Outage
We're back online.

Outage was due to a faulty router at the data center.

Statistics for August 01, 2009 through August 31, 2009
New Submissions: 128832
-General: 101100
-Mature: 14477
-Adult: 13255
New Journals: 46429
New Comments: 1459135
New Users: 11417
Active Users: 91547

Total Advertising Impressions: 251,801,127
Total Ads Clicked: 125,888
Most Clicked Ad: Smoking Pen
Note that the ads only ran from August 8 onward.

(And since I apparently forgot to post it to LJ last month...)
Statistics for July 01, 2009 through July 31, 2009
New Submissions: 127916
-General: 99819
-Mature: 14878
-Adult: 13219
New Journals: 45170
New Comments: 1410413
New Users: 12203
Active Users: 68454

(Comment on this)

Fur Affinity Everything Outage 2009-09-01
There was a four hour outage last night due to routing issues at Ironpath.

The site is down now due to routing issues at Ironpath.

No ETA.

(3 Comments |Comment on this)

Monthly Statistics: June 2009
Statistics for June 01, 2009 through June 30, 2009
New Submissions: 107450
-General: 83141
-Mature: 12928
-Adult: 11381
New Journals: 39089
New Comments: 1195527
New Users: 10233
Active Users: 52152

Delayed due to Anthrocon.

Because of the way the active user count works, it's only accurate to do the count right as midnight on the first of the month, which I didn't do. All the other stats are fine, though.

(1 Comment |Comment on this)

Forum Outage 2009-06-30/07-01
Forums are out due to MySQL crapping itself. (I/E: Running out of disk space.)

They'll be back online sometime in the next hour or two.

(Comment on this)

Mainsite outage about an hour ago

(Comment on this)

Hardware Failure
Ghostlight, FA's Forum VPS hosted by Rackforce, has experienced catastrophic failure resulting in a total loss of data. The RAID card was the point of failure. Rackforce is not sure if they will be able to recover as of this point.

We do have a backup, and will be attempting to restore services as soon as we can. There is no ETA as of yet.

(5 Comments |Comment on this)

Monthly Statistics: May 2009
Statistics for May 01, 2009 through May 31, 2009
New Submissions: 97155
-General: 74219
-Mature: 12022
-Adult: 10914
New Journals: 34852
New Comments: 1088683
New Users: 9607
Active Users: 64738

Not doing ad stats this month due to the issues we've been having of late. Everything should be back on track next month.

(Comment on this)

Monthly Statistics: April 2009
Statistics for April 01, 2009 through April 30, 2009
New Submissions: 91822
-General: 70583
-Mature: 10974
-Adult: 10265
New Journals: 32899
New Comments: 1019193
New Users: 9942
Active Users*: 61233


Ad Impressions: 194,132,873
Ads Clicked: 96,477
Most Clicked Ad**: Cocktails
* Take this stat with a grain of salt. I waited too long...
** Damn it, furries. Stop proving me right.

As they say, better late than never...

---

And I know I've been utterly horrible about posting downtimes here. Real life awesomeness. Most of this months downtime and slowness was due to an update to the ad serving software that completely broke our MySQL optimization.

(1 Comment |Comment on this)

Monthly Statistics: March 2009
Statistics for March 01, 2009 through March 31, 2009
New Submissions: 95656
-General: 72863
-Mature: 11865
-Adult: 10928
New Journals: 33011
New Comments: 1054400
New Users: 9804
Active Users*: 69988
Data Transferred**: Not available.

Ad Impressions: 264,868,891
Ads Clicked: 126,811
Most Clicked Ad***: Cocktails

Total Donations: Pending...
Total Ad Revenue: Pending...

* Take this stat with a grain of salt. I waited too long...
** Yes, awstats still can't cope with our logs. It's on the "to do" list.
*** Damn it, furries. Stop proving me right.

Bah. I suck. That whole "exam" thing I was doing this last week kinda got in my way.

On another note:

Whenever someone makes a post to the community, I see in my inbox: "Moderated submission notification." And that is all I see before I click the check next to it and head up to the "Archive" button. Because 99.9% of them are "hello" "commissions 4 sale" and "check out my auction."

Now, I'm not opposed to posting other things here that fall within the community rules. But please come talk to me about it, or I probably won't even notice.

(Comment on this)

Monthly Statistics: February 2009
Statistics for February 01, 2009 through February 28, 2009
New Submissions: 89926
-General: 67829
-Mature: 11401
-Adult: 10696
New Journals: 30823
New Comments: 997014
New Users: 8217
Active Users: 55847
Data Transferred: Not available.

Ad Impressions: 242,067,927
Ads Clicked: 81,463
Most Clicked Ad: Bad Dragon

Total Donations: $939.74 across 52 donations.
Total Ad Revenue: $510

This month only:
"yak is awesome" donations: $575.13 across 67 donations.

(Comment on this)