You are viewing furaffinity

Furaffinity Status' Journal
 
[Most Recent Entries] [Calendar View] [Friends]

Below are the 20 most recent journal entries recorded in Furaffinity Status' LiveJournal:

    [ << Previous 20 ]
    Tuesday, December 21st, 2010
    8:12 pm
    [murasadramon]
    December 2010 Hacking
           By now, many of you know that Fur Affinity was attacked on Thursday, December 16th 2010. Attackers were able to compromise the admin system using an XSS exploit in the trouble ticketing system to gain control of an admin account. We pulled the website offline, and closed the hole that lead to the initial attack, but not before the intruder was able to illegally compromise the private notes of 41 users (including admins and staff) and the vandalism of several galleries. Regrettably, the leeching of notes occurred before the hacker made his presence known on the site, and we were not able to stop it.

           At no point were user passwords or the site database compromised.

           After closing the initial hole that the intruder was using to compromise the site, they then attacked an admin's e-mail, managing to compromise their email account to perform a password reset. With the new password, they were able to get back into the site and into the forums. At the same time, another attack was launched on a second admin, compromising a long-abandoned account they had which was setup as an e-mail fallback for their main account. In both instances, the attackers were able to gain access back into the system, causing scattered vandalism.

           We were able to flush the attacker out of the system through multiple wipes of cookies and active login sessions (which some of you may have noticed when your account were logged out).

           After Yak revised and recoded the security side of the admin panel, the attackers then launched a distributed denial of service (DDOS) attack against FA as a final measure. Working with our host, we were able to block the attack and restore services to the site. While we had initially suspected potential issues due to the 1.2 million Gawker passwords that were leaked (which had affected some regular users on the site), we want to clarify that the Gawker leaks WERE NOT an issue with the FA intrusion.

           Galleries which were wiped are in the process of being restored, and we are working to strengthen and improve security. We have also removed the ability for certain admins to view notes. We will be bringing in additional coding help to perform security audits and improve upon the site's platform, as we do take security seriously. We regret that this happened, and ultimately the blame for this lay with us for letting the hole slip through the cracks. That said, it does not excuse the intruders for their actions, and we are working with law enforcement to pursue the issue.

           On behalf of the entire staff of FA we apologize for what happened. We make no excuses for what happened.

    ----------------------------

           If you have questions, please feel free to ask, we will update the thread with a Q/A. Keep responses civil, and honest. There has been enough drama over this, and we want to work towards peaceful resolution.

    EDIT: Had the date wrong in the initial attacks. My apologies. The initial incident happened on Thursday, not Friday.
    Wednesday, December 9th, 2009
    3:23 am
    [thevirtualcat]
    Site back online.
    You probably knew that already, though.
    Tuesday, December 8th, 2009
    1:38 pm
    [thevirtualcat]
    Forums are back online.
    Thanks to Carenath donating space on his server, the forums are back online.

    Now, I will preemptively answer some questions.

    • This wasn't done earlier because I didn't have the data off the old forum server until late, late Sunday night.
    • No, we can't use Carenath's server to bring the mainsite up. The main site is about a thousand times larger than the forums. (Both in terms of disk space and bandwidth requirements.)
    • We have everything we need to install the servers at the new facility. The only concern now is weather related, but that shouldn't be a problem. (Incidentally, did you know that studded tires are illegal in Maryland? This is highly relevant.)
    Monday, December 7th, 2009
    5:33 pm
    [murasadramon]
    Good News, Bad News, Good News, Bad News
    Good News: We're all ready to go on our end. We have the IPs, we've got the rack setup at our new colo. All we need to do is move our servers into the facility, update the IPs and config the DNS. That'll take maybe 30 to 45 minutes.

    Bad News: Switch and Data (hey, remember them?) were not able to process our badges in time, so we're not able to get into the colo. Yet. Fuck. You may proceed to queue the "BADGES?! WE DON'T NEED NO STEENKIN' BADGES!" quotes, because... that's the only thing holding us up. THE ONLY THING. Only thing. Yep. That. That thing? That's the only thing. THE. ONLY. THING.

    Only thing.

    Good News: Net-cat and myself (and maybe Dax) will be meeting at the colo at 5:00PM EST tomorrow to setup, install, configure and bring Fur Affinity online. We had hoped to do this today, but Murphy's Law is a fickle bitch unfortunately the paperwork pushers didn't pushpushpush.

    Bad News: I'm out of Cherry Bawls. FFFFFFFFFFFFFFFFFFFFFFFFFFFF

    Current Mood: annoyed
    12:55 am
    [murasadramon]
    FA Status Update
    Just a note:
    Net-Cat drove over to my place tonight and we did pre-maintenance on FA's servers (Tiamat, Bahamut, Trogdor and Novastorm) and all systems are good to go. We updated some of the software on Novastorm (VMWare) to the latest version.

    As of right now, everything is solid on our end. We are currently waiting for the colo facility to process our paperwork and get the billing process started. That's our only hold up right now. :) No, we do not have an ETA at this time, but we're still on track for our original forecasts. Again, everything is good to go on our side.

    At this point it's all paperwork.
    Friday, December 4th, 2009
    2:53 pm
    [murasadramon]
    It looks like we're planning to bring the site up Monday night. While we were originally aiming for this weekend, we're currently waiting on the paperwork to process at our new colo facility. While the tech team works 24/7 the back office team is a 9-5 crew. This may turn out to be a good thing as they are expecting snow, and the first snow of the season can usually be kind of hazardous road condition wise. =P

    We'll keep you updated as usual.

    On the plus side here's the good news: we're upgrading to a 100Mbit dedicated line, which is roughly 25% more bandwidth. In addition, our new deal allows Fur Affinity to operate its main data center at roughly 40% cheaper costs. This will allow us to further invest money into better hardware, upgrades and permit us to do off-site backups.

    (Note: With all due respect to Canada, our new colo facility shall remain in Virginia.)
    12:16 am
    [murasadramon]
    Fur Affinity will be returning [this space intentionally left blank]*.

    * Update shall be later tomorrow to fill in the gap

    ---------------------

    I was able to down to the root of things. The entire outage goes down to a hardware problem on XO Communications. XO Communications did not have replacement hardware for the fiber lines that went down, and had to order it. Much like Comcast, when they say "we'll have a technician there at 2pm" and by the time 6pm rolls around he hasn't shown up, XO Communications dropped by ball.

    The entire issue was supposed to have been fixed by Tuesday night. This entire outage can, should and will be blamed on them. I'm not going to excuse the total lack of communication on our host's part, but that's another story altogether.

    I sincerely apologize for the outage, the inconvenience and frustration this has caused. This is not how we want FA to be known, and we've been striving to improve everything, and we'll be doing just that in the coming months.

    I have more I could say, and say venomously, but I'm going to hold my tongue. I've been working on FA and pulling servers up until midnight. I'm tired, and shall retire to bed. For now, adieu.
    Wednesday, December 2nd, 2009
    11:33 am
    [murasadramon]
    Site Status
    Just as an FYI - as I posted to the FA Twitter, we have put out a few quotes and bids on alternative hosting for other colos. We find the current situation to be unacceptable, and do not approve of how things are being handled.

    While we we do not have new information at this time, we're checking into and examining all available options.

    Update 1 - 02DEC@12:01 EST
    We've found a solid co-location facility in Montreal, Canada. I am waiting to hear back from some other quotes at this time.

    Update 2 - 02DEC@13:55 EST
    Talking to two other colos - one of which is in walking distance from my apartment. They local one is offering a deal that may be "too good to be true" due to recent expansion (they're trying to fill space ASAP, so the deal may be legit) but we're investigating it. It *is* from a legit company that's been in business for 10+ years, so... we'll check it out.

    More detail as follows. We're not rushing into this, and we want to do things right.

    Update 3 - 02DEC@18:41 EST
    I have a scheduled tour of a new colo tomorrow. Keep your fingers crossed. If it's as good as everything appears to be be, this will not only be a professional colo/hosting solution (with a promised 99.9% uptime backed by an SLA) but will save Fur Affinity quite a bit of money annually.

    We will keep you informed.

    NOTE: I know we're even not three days into this outage (Outagegate), but I would like to re-iterate one thing: this will NOT be an issue like last August. We will not need donations, nor ask for them. If anybody tells you otherwise you have our permission to slap them.
    Monday, November 30th, 2009
    12:30 pm
    [thevirtualcat]
    [Updated 2009-12-02 02:00 UTC] Oh, and you thought your cable outages were annoying?
    Imagine how we feel when our ISP, which supposedly has redundant links, seems to have perennial peering issues.

    Pretty much the only way you're going to be getting your wholesome, furry goodness *cough* at this point is by throwing a party in the server room. (And I'd imagine that S&D would take issue with that.)

    tl;dr - ISP's shit's broke. Nothing we can do about it. Patience is a virtue. Etc, etc, etc.

    PS: I totally just noticed that I haven't done stats in like... two months. Not that I can do it now. Servers are inaccessible. :D

    Lack of update: 2009-12-01 15:30 UTC (10:30 EST)

    Well, what can I say. Our ISP is still not talking to us. (And I'm sure anyone who is resourceful can dig up who our ISP is.)

    Going on 24 hours with no updates. When we know something, we'll post it.

    We also have no plans to ask for donations for this problem. Because it won't actually solve the problem, in this case.

    Update: 2009-12-02 02:00 UTC (21:00 EST)

    (08:48:04 PM) Dragoneer: That the fiber lines were damaged, they're doing emergency repairs.

    No estimate.
    Monday, September 14th, 2009
    1:23 pm
    [thevirtualcat]
    More routing issues. 2009-09-14
    We've had reports of people outside the US having trouble accessing the site.

    This is due to routing issues at an upstream provider.

    Cages have been rattled. All we can do now is wait.

    Sorry for the inconvenience.
    Tuesday, September 1st, 2009
    1:40 pm
    [thevirtualcat]
    Monthly Statistics, Outage
    We're back online.

    Outage was due to a faulty router at the data center.

    Statistics for August 01, 2009 through August 31, 2009
    New Submissions: 128832
    -General: 101100
    -Mature: 14477
    -Adult: 13255
    New Journals: 46429
    New Comments: 1459135
    New Users: 11417
    Active Users: 91547

    Total Advertising Impressions: 251,801,127
    Total Ads Clicked: 125,888
    Most Clicked Ad: Smoking Pen
    Note that the ads only ran from August 8 onward.

    (And since I apparently forgot to post it to LJ last month...)
    Statistics for July 01, 2009 through July 31, 2009
    New Submissions: 127916
    -General: 99819
    -Mature: 14878
    -Adult: 13219
    New Journals: 45170
    New Comments: 1410413
    New Users: 12203
    Active Users: 68454
    11:43 am
    [thevirtualcat]
    Fur Affinity Everything Outage 2009-09-01
    There was a four hour outage last night due to routing issues at Ironpath.

    The site is down now due to routing issues at Ironpath.

    No ETA.
    Thursday, July 9th, 2009
    12:29 pm
    [thevirtualcat]
    Monthly Statistics: June 2009
    Statistics for June 01, 2009 through June 30, 2009
    New Submissions: 107450
    -General: 83141
    -Mature: 12928
    -Adult: 11381
    New Journals: 39089
    New Comments: 1195527
    New Users: 10233
    Active Users: 52152

    Delayed due to Anthrocon.

    Because of the way the active user count works, it's only accurate to do the count right as midnight on the first of the month, which I didn't do. All the other stats are fine, though.
    Wednesday, July 1st, 2009
    1:19 am
    [thevirtualcat]
    Forum Outage 2009-06-30/07-01
    Forums are out due to MySQL crapping itself. (I/E: Running out of disk space.)

    They'll be back online sometime in the next hour or two.
    Friday, June 19th, 2009
    2:50 pm
    [thevirtualcat]
    Mainsite outage about an hour ago
    Originally posted by yak, on the forums
    ... was caused by running out of hard drive space on the database server.

    The server actually has plenty of storage for it's needs, just that some of that storage was temporarily occupied by database backups and copies for development purposes. After restoring the ads on the mainsite, openx' database began growing in size explosively until eventually it used up what little free space there was on the server.

    That was actually a very simple, stupid problem and the reason it took so long to fix was because of the completely misleading behavior of the servers. All the time I was thinking the problem was with the application server, and looked for potential problems there.

    Anyway, the problem was resolved and FA is back online now.
    If you are still seeing the offline page, please clear your browser's cache.
    Sunday, June 14th, 2009
    2:39 pm
    [murasadramon]
    Hardware Failure
    Ghostlight, FA's Forum VPS hosted by Rackforce, has experienced catastrophic failure resulting in a total loss of data. The RAID card was the point of failure. Rackforce is not sure if they will be able to recover as of this point.

    We do have a backup, and will be attempting to restore services as soon as we can. There is no ETA as of yet.
    Friday, June 5th, 2009
    12:29 am
    [thevirtualcat]
    Monthly Statistics: May 2009
    Statistics for May 01, 2009 through May 31, 2009
    New Submissions: 97155
    -General: 74219
    -Mature: 12022
    -Adult: 10914
    New Journals: 34852
    New Comments: 1088683
    New Users: 9607
    Active Users: 64738

    Not doing ad stats this month due to the issues we've been having of late. Everything should be back on track next month.
    Monday, May 4th, 2009
    11:56 am
    [thevirtualcat]
    Monthly Statistics: April 2009
    Statistics for April 01, 2009 through April 30, 2009
    New Submissions: 91822
    -General: 70583
    -Mature: 10974
    -Adult: 10265
    New Journals: 32899
    New Comments: 1019193
    New Users: 9942
    Active Users*: 61233


    Ad Impressions: 194,132,873
    Ads Clicked: 96,477
    Most Clicked Ad**: Cocktails
    * Take this stat with a grain of salt. I waited too long...
    ** Damn it, furries. Stop proving me right.


    As they say, better late than never...




    And I know I've been utterly horrible about posting downtimes here. Real life awesomeness. Most of this months downtime and slowness was due to an update to the ad serving software that completely broke our MySQL optimization.
    Saturday, April 4th, 2009
    11:58 pm
    [thevirtualcat]
    Monthly Statistics: March 2009
    Statistics for March 01, 2009 through March 31, 2009
    New Submissions: 95656
    -General: 72863
    -Mature: 11865
    -Adult: 10928
    New Journals: 33011
    New Comments: 1054400
    New Users: 9804
    Active Users*: 69988
    Data Transferred**: Not available.

    Ad Impressions: 264,868,891
    Ads Clicked: 126,811
    Most Clicked Ad***: Cocktails

    Total Donations: Pending...
    Total Ad Revenue: Pending...

    * Take this stat with a grain of salt. I waited too long...
    ** Yes, awstats still can't cope with our logs. It's on the "to do" list.
    *** Damn it, furries. Stop proving me right.


    Bah. I suck. That whole "exam" thing I was doing this last week kinda got in my way.

    On another note:

    Whenever someone makes a post to the community, I see in my inbox: "Moderated submission notification." And that is all I see before I click the check next to it and head up to the "Archive" button. Because 99.9% of them are "hello" "commissions 4 sale" and "check out my auction."

    Now, I'm not opposed to posting other things here that fall within the community rules. But please come talk to me about it, or I probably won't even notice.
    Monday, March 2nd, 2009
    8:03 pm
    [thevirtualcat]
    Monthly Statistics: February 2009
    Statistics for February 01, 2009 through February 28, 2009
    New Submissions: 89926
    -General: 67829
    -Mature: 11401
    -Adult: 10696
    New Journals: 30823
    New Comments: 997014
    New Users: 8217
    Active Users: 55847
    Data Transferred: Not available.

    Ad Impressions: 242,067,927
    Ads Clicked: 81,463
    Most Clicked Ad: Bad Dragon

    Total Donations: $939.74 across 52 donations.
    Total Ad Revenue: $510

    This month only:
    "yak is awesome" donations: $575.13 across 67 donations.
[ << Previous 20 ]
FurAffinity   About LiveJournal.com