We've just had our first story posted that exceeded the size that Shifti could handle in a single page; Michael Bard's 737000-byte "Mythic Journeys", now split up into two separate pages for your convenience. I consider that to be an interesting landmark of a sort.
We've just had our first story posted that exceeded the size that Shifti could handle in a single page; Michael Bard's 737000-byte "Mythic Journeys", now split up into two separate pages for your convenience. I consider that to be an interesting landmark of a sort.
:::: [[User:Bryan|Bryan]] 04:47, 28 February 2008 (EST)
:::: [[User:Bryan|Bryan]] 04:47, 28 February 2008 (EST)
:After checking the software and the error message that Bard saved I can safely report that he managed to hit a bug in the MediaWiki software itself and the way the parser works. A 737000-byte story without any "extension" tags (like "poem") would not have hit this problem. In fact, the MediaWiki software itself has a hard limit of 2048 kilobytes — that's 2 megabytes — on article size. What happened in this case is that the article text got duplicated and stored in enough places that the software overran the 40 megabytes that the software has been allocated.
:This problem cannot be fixed—it would require redesigning and rebuilding the entire parser from scratch. It can be averted—by asking that individual stories are kept under 400 kilobytes. Now the more technically astute of you may be wondering why I can't just increase the memory allocation. In truth, I could. But that would be a stopgap measure, because the available ram on the server is limited. Until I upgrade it the server is limited to the 768M of ram installed in it, and that is in use by Apache, MySQL and my e-mail server. To be fair, the server is old—just how old I'm not revealing, but suffice to say it started life as a desktop and got repurposed.
:::::[[User:ShadowWolf|ShadowWolf]] 22:51, 28 February 2008 (EST)
Notification banner removed — we've been up for more than 24 hours on the new router and there has not been a problem.
Notification banner removed — we've been up for more than 24 hours on the new router and there has not been a problem.
Revision as of 04:47, 15 March 2008
Well, it's been one week and we're almost halfway to our goal of $300. Sadly, the donation drive ends at 2359 EST tonight and I don't think we're going to make that. However, we've got enough to make the move and go for a month or two. This doesn't mean that we're not going to move. Quite the contrary, we're going to move so we can more easily handle the amount of traffic we've been getting.
In other news... Happy Birthday to Me! Yep, at 0000 Saturday, March 15 I turned 30. In related news, this birthdate is also shared by Phil Geusz, but I'm not sure how old he is.
After that OOM episode on the fifth, I started looking at what options existed for high-availability hosting for Shifti. After much discussion and weighing of risks I still hadn't been able to make up my mind as to what to propose to the other admins. So I talked to Viqsi about the options and she agreed that the best option would be to get two low-cost servers and then split the load between them.
What this means for Shifti is that, depending on the currently running donation drive, the site will be on its new servers no later than the seventh of April. What this means is that there will probably be a period of downtime, hopefully not more than two hours, while the database is backed up, transferred to the new server and restored. If you'd like to see Shifti make the move to the new servers faster, you could always donate some money—there is a link in the nifty banner at the top of each page.
And there we go... Not even five minutes ago the server went "OOM" (that is, it ran out of memory). This is because the server is an old system, mostly. It's a P3/800 with 768M of RAM. If I had the space in the box I'd install another hard drive and allocate more swap space, but...
While Jon Buck has graciously offered to donate his old computer as a new server the problem is that even that will probably not help the problem much. But just replacing the server will be a stopgap measure, since we are approaching the limits of even the upgraded bandwidth that the readers donations have paid for. Tonight I will be discussing a complete move to a server hosted by ServerBeach with the rest of the administrators and that will probably lead to another ugly donations banner.
There is apparently enough traffic that MySQL's setup - which was already set at double the defaults - started having all kinds of problems. So I've gone and doubled it's settings yet again. At the same time I've done some poking at Shifti's configuration and upped quite a few of MediaWiki's internal settings. Hopefully this will make the site a bit more stable, but if things don't work out, we will have to start planning a move to ServerBeach.
Shifti has reached a level of traffic that I am not sure the server will be able to continue to manage. We are reaching anywhere from 80 to 100 megabytes of bandwidth use a day - an average of more than 200 unique visitors and 10,000 hits per day. Truthfully, the server will probably scale up to more than the current level, but the default for Apache is a max of 20 simultaneous requests and I have changed that to 50.
In other news, after all the work he did to get it into a state where it wouldn't cause the server to choke, we have had to delete the "All In One" version of Michael Bard's "Mythic Journeys" because it was causing some parts of the system to slow down during database accesses.
Anyway, it's time for me to go back to making sure that Shifti stays online and stable. :)
We've just had our first story posted that exceeded the size that Shifti could handle in a single page; Michael Bard's 737000-byte "Mythic Journeys", now split up into two separate pages for your convenience. I consider that to be an interesting landmark of a sort.
After checking the software and the error message that Bard saved I can safely report that he managed to hit a bug in the MediaWiki software itself and the way the parser works. A 737000-byte story without any "extension" tags (like "poem") would not have hit this problem. In fact, the MediaWiki software itself has a hard limit of 2048 kilobytes — that's 2 megabytes — on article size. What happened in this case is that the article text got duplicated and stored in enough places that the software overran the 40 megabytes that the software has been allocated.
This problem cannot be fixed—it would require redesigning and rebuilding the entire parser from scratch. It can be averted—by asking that individual stories are kept under 400 kilobytes. Now the more technically astute of you may be wondering why I can't just increase the memory allocation. In truth, I could. But that would be a stopgap measure, because the available ram on the server is limited. Until I upgrade it the server is limited to the 768M of ram installed in it, and that is in use by Apache, MySQL and my e-mail server. To be fair, the server is old—just how old I'm not revealing, but suffice to say it started life as a desktop and got repurposed.
New router has arrived, been installed and setup. I can't, in good conscience, clear the trouble flag floating as a page banner yet, but hopefully all the problems that we've been having with the connection are now gone.
The router is by no means stable, but it is much more stable than it has been. We are now seeing periods of almost exactly 24 hours between required reboots of the router. Hopefully the new one (scheduled to show up on 2/19) will correct that final problem. In the meantime the banner will remain so people are informed that the site may disappear out from under them.
Once more the router surprises me and remains stable throughout the day. I've checked with some trusted friends (Thank you Chloe!) that are also highly-paid, self-employed network specialists and the information I have on my connection seems to indicate that, while it does have a somewhat high "Signal Attenuation" (44 to 45 dB) and a low "SNR Margin" (18 to 19 dB) there are no other real problems. Although there still might be an intermittent short somewhere in the houses wiring causing these problems, the predictable period they had and the fact that the problem seems to be clearing up on its own pretty much rules that out. Anyway, the 24 hour clock on "Is the problem gone" is running again, and if, by 1100 EST 2/10 the problem hasn't re-occurred, I'll see about removing the emergency notice banner.
It appears that my restraint in not removing the banner was well worth it. It went and bombed again sometime in the seven hours I was asleep. But from 1700 Eastern 2/8 to at least 0300 Eastern the next day — a period of about 10 hours — it ran like it used to. If I could figure out what has occurred to create that period of uptime I'd know who to thank. Anyway, I'm a bit annoyed that the problem that appeared to have fixed itself hasn't actually been miraculously fixed. However, I am happy that my speculation that it was a DDOS is incorrect… Well, I can't actually say it isn't a DDOS against Shifti — it could still be such, but the evidence doesn't really support the diagnoses.
This might be the time to start cheering and jumping for joy, folks. I last had to reset the router at 2209 UTC, according to the routers internal logs. That is, I reset it at 5:09 PM EST and it's run without a hickup since. Not that the problem is definitively gone, but I've started going through the piles of traffic logs here and we either got hit with a DDOS that wasn't detected and logged as such by the router (not surprising, it wasn't designed to pick up on them, just normal DOS attacks) or my ISP was being a complete and total ass, having problems at their end and blaming the end-user.
Since it's now 0400 UTC on the next day and the router hasn't died like it has been, I'm going to conditionally declare the problem over. However, moving to ServerBeach is still probably the best bet, since a host there will have a better connection and be better able to survive a DDOS and other potential problems. Until I have clear proof that the router isn't going to drop (like maybe having it continue functioning for a full 24 hours) the banner will remain.
However, moving Shifti to ServerBeach is not free — the service is a MINIMUM of $75US per month, and while I could probably cover that expense without any problems, it will be better for Shifti's future to not rely on a single person for funding and rather just fund itself. So please, if you have the money to spare, Donate.
On examining available evidence it's increasingly looking like the problem isn't that the routers hardware is dying, but that the router I have isn't able to handle the volume of traffic the site is getting. So the admins have been discussing moving the site to a hosted service that provides dedicated servers for all of their customers. The service we are looking at, as one possible solution, is provided by the same company that hosts transform.to and was recommended by none other than the Dragon DeMonsyne. The reason we are looking into this, rather than taking up the offers of several people that have servers sitting on very good connections is that, by putting Shifti on its own server like this, we have a way to keep Shifti online despite what may happen to any single person. If (when?) Shifti does move to the new host I'll still be around, as I am the resident code-monkey as well as the host.
In other news I'm still fighting off the bronchitis. I thought I'd beat it and only had a cold left, but I've been getting chills and other signs of a lingering fever, so... Anyway, when I'm back at more than the 60% health I'm running at currently I'll work on putting a script together to reset the router every sixty minutes to attempt to keep things fully functional until the new router shows. This will probably negatively impact my connectivity, so I also might attempt doing a "direct bridge" of the router/modem with the server and running all my connectivity off the server for a while.
In other words, lots of options on how to handle things. Sadly, I can't seem to figure out a simple way to script the router reset, but that doesn't mean I'm not going to go ahead with it.
The Router will not be arriving until 2/19 — apparently they forgot to update their website and the router went out of stock. In other news Moms has pneumonia and me and pops both have bronchitis. Oh Joy of Joys!
Bad news, everybody. The Router did not ship today like it should have. I don't know why, but I'm going to be calling the company as soon as I can get up… Yes, the bronchitis that Moms had has spread to everyone — where it was just a bad cold on the first I now have a low-grade fever and a full body ache. But I will make it to my feet — if I don't there will be a mess in the living room. Anyway, just felt everybody should know, and now it's time for me to get moving.
Everyone in the house here is ill, so I couldn't make it to a physical store today. But never fear — I ordered a new modem/router online and requested FedEx 1-Day shipping, so by Tuesday, at the latest, Shifti will have a new router managing it's internet connection. As a bonus the new modem/router incorporates a wireless hub, so I will be able to ditch the dinky, pointless Belkin wireless hub I've been using. If anyone want's a two year old Belkin Wireless-G hub (model F5D7230) I'll gladly see about boxing it up and shipping it.
The modem/router managing Shifti's internet connection appears to be dying, since all other potential causes of the sporadic downtime have been taken into account. But never fear, for I will be purchasing a new and much better model after I wake up tomorrow. To everyone hampered by these problems, I am sorry, but the router has faithfully worked non-stop for the past two years and I had no reason to suspect it would not continue doing so. Hopefully by this time tomorrow Shifti will be running better than ever. In the meantime I am going to be around as much as I can to keep resetting the router as it becomes necessary.
It appears that our boost in traffic is thanks to a number of factors, but the largest of them appears to be that Felix, Mr. Peaches and several others have stories listed in the Ursa Major Recommended Furry Art and Literature List.
On another note we've had a few more spammers and I've just caught someone re-ordering the Story Archives page to give favored placement to one site or another. I've already said my piece over at the Talk page, so I won't repeat it here.
Bandwidth upgrade completed a day or two back. Shifti is now on the best connection the server's location is capable of. If the bandwidth use keeps going up we'll be left either needing to move the server to a colo facility or playing tricks with DNS to split the load across a number of servers. That is still an option, if we can find a way to keep the database available even when the system is being bombed. (I will be looking into this!)
Have you ever wished you could write stories like those you find on Shifti? Or do you wish that your skill as a wordsmith was even better than it currently is? Fret and and wish no more, the Administrators have heard your pleas and have added tools and information to Shifti in response. That's right—Shifti now has a Writer's School!
The donation money has been well used. Within a week Shifti's connection will have something like thee times the bandwidth it currently does. On another note, for the first half year of Shifti's existence it used about 30M of bandwidth a day. In December that figure jumped radically, along with all the other indicators. This month, January of 2007, Shifti has already used over 400M of bandwidth and has had almost two-thousand unique visitors. What that means is that Shifti is a success—thank you to all the authors who have begun uploading content, and thank you to all the people that have advertised Shifti for us.
In other news Shifti has added another Administrator. Devin has been with us from the start and, even without any official status, has championed Shifti. Thanks to Devin stories have started to be imported from the old Transformation Story Archive—his favorites list began as a set of links to his favorite stories in the TSA—and that started the first round of stories being brought over. More recently he has contributed to the new "Pig and Whistle" universe by taking part in the still in progress Collaborative Story and marking the universe as a something he'd like to see succeed. But more than that he has been, to coin a phrase, Shifti's biggest cheerleader. In recognition of all of his contributions to this site, the existing Administrators decided to honor him with Administrator status.
During the late night of January 3 and early morning of January 4 some strange numbers showed up in Shifti's logs. When examining the log files by hand I found the cause – one person was somewhat monopolizing all bandwidth use on Shifti. This person had pointed the automated HTTP and FTP enabled download agent 'wget' at Category:Story in an attempt to download a copy of all the content on Shifti. That action, though legal, isn't very nice, since the bandwidth it uses has to come from somewhere and the place it comes from is other users of Shifti. For about an hour I had the server returning the HTTP "Forbidden" error (403) to any software reporting as wget. However, after discussion with the other administrators it was decided that the 403 error was a bit heavy-handed. While the error is now gone, there are other options that are being explored. (In fact one of them caused a 10 or so minute loss of connection the other night o.O)
Okay, I know it's ugly and people are going to complain, but until I'm positive Shifti can survive with people going through the donations page the banner that now starts every page is going to remain. While I (ShadowWolf) thought that I'd be able to keep Shifti online without any problems, the scare last night has shown me that donations are needed to make sure that Shifti stays online. So please donate and keep this increasingly valuable resource online!
Your friendly neighborhood server owner almost lost all internet connectivity due to unforeseen difficulties with cash flow today (Jan. 3, 2008). To stop this from ever happening again, I've setup the Donations page to point to my PayPal account. The money will be used to cover connectivity fees and also to provide upgrades to the servers hardware. So please, donate today!
Old News 
|
Old News 
