Web Hosting Services
Order Web Hosting
Web Hosting Tutorials
Web Hosting FAQs
Reseller Hosting
Web Hosting Affiliate Program

1999 Reports

2 December 1999

On Wednesday morning, AT&T Worldnet did something to their DNS servers that resulted in a complete failure of their entire network. All 1.9 million dial-up subscribers are off-line, unable to reach any part of the internet -- and that includes checking email on our system. As of this morning, service has not been restored and AT&T has no time estimate of when that service will be restored. Please note that PagePlop does not use AT&T for its backbone connectivity and is not affected by the AT&T outage.

Any of our clients who are affected and are unable to retrieve email, send email, or make alterations to their web sites will either have to wait it out or use an alternate dial-up provider to do any work necessary. In the meantime, I will be more than glad to rely email messages, open client email and read it to them over the phone, or make changes to their web sites as needed until this outage is over. Unfortunately, there is little I will probably need to actually do since no one who is using AT&T and affected by the outage can read this page to note my offer.

But it's a nice gesture. :)


27 November 1999

I just wanted to put up a note on what I think is going to happen as the result of Y2K with respect to your web sites serving properly.

There are absolutely no issues involving Y2K problems with respect to the PagePlop system, it's hubs, switches, or routers, or its connection to the backbone of the Internet. BellSouth, Sprint, and Carolina Power and Light have assured us that their entire systems are fully ready to go and have been repeatedly tested. That does not mean that there will not be issues to deal with, however.

I anticipate that most of the world will be picking up their home phones shortly after midnight to check if they still have dial tone. If too many of them do so, they will indeed not have a dial tone. They will find the same conditions as they do on Mother's Day or Christmas morning when the phone systems are overwhelmed. That means that if you need to call us, you probably will not be able to do so until things settle down. The voice phone system is not the same as the Internet-connectivity system, though, so our digital lines are not affected at all.

But everyone will also try to hit their web site at the same time to see if we and other providers are still serving. That will probably result in a massive overload of the Internet backbones, though PagePlop is fully capable of handling just about any traffic you can throw at us. Even if every one of our clients hit their home pages at exactly the same time, we estimate it would take less than 20 seconds to clear out the requests.

So the question becomes, if I can't get to my website because the Internet is jammed and I can't call because my phone is jammed, then how do I know what is happening? Well, you don't. All you can do is sit tight and wait for folks to eventually realize that there is no problem. They will stop picking up their phones and they will stop trying to hit web sites and things will calm down.

Now, the more serious issues...

Since there are no major points of failure anywhere in the telco or power systems, I would anticipate that more than a handful of ignorant cretins will attempt to cause havoc by introducing viruses or denial of service attacks against the more high-profile web sites and the backbone providers. They are all aware of the potential of malicious intent and will be at the ready to intervene. That does not necessarily mean that there will be no significant problems, though. It simply means that any problem that arises will not be the result of a Y2K problem.

Then there is the water supply. I anticipate a very large, but temporary problem with the water supply especially in major cities. What I think will happen is that folks will be curious as to whether the water is still on and flush their toilets or turn on faucets pretty much at the same time. The resulting drop in water pressure will be similar to what occurs at half-time of the Super Bowl when the entire nation flushes simultaneously.

I also anticipate runs on cash, food, and medicine which will deplete banks, grocery stores, and pharmacies nationwide. Unfortunately, I fear that a synergy will set in. In other words, the shelves will be sparse so folks will think the problem is with the food distribution system and horde even more. What we will end up with are a whole lot of households which contain three months worth of toilet paper and green beans for absolutely no reason. That will also result in a situation where folks for weeks or months will spend less than normal throwing off store buying patterns. We may even end up in a small recession as consumer spending drops and interest rates rise because of increased demand by businesses for short-term, cash-flow loans in the first quarter of 2000, loans they would not need if people just maintained their normal buying patterns.

I can also envision a dramatic spike in crime, particularly burglaries. Think about it. People start taking money out of their accounts and the only place they have to keep it is at home or work. Then they will go out to party leaving their homes and offices without protection. That is an open invitation for the scum of the earth to break in and take goodies. What concerns me most about that scenario is when burglar alarms start sounding at police stations and they either assume that it is a Y2K glitch or they are so overwhelmed with alarms that they can in no way respond to them all in a timely manner. As soon as (or if) criminals figure out that the police are overwhelmed, that pretty much gives them an open season on committing further crime like looting. It may not be very pretty until the sun comes up on January 1st.

But it won't have a thing to do with any direct computer problems, just human stupidity.


13 October 1999

FTP access is currently turned off for approximately 100 clients because we are having some problem with one particular server. We are afraid that the latest version of MGI is somehow conflicting with the FTP server and are trying to identify the exact cause. I anticipate that FTP access will be turned back on about 2 PM EDT.


21 September 1999

Things are completely back to normal at least in Raleigh. If you go not 20 miles east of us, you find a 120 stretch miles of the worst flooding in North Carolina history. Whole towns are under water (like 30 feet of water,) Interstates are closed, and there is a tremendous worry about disease since some 500,000 hogs, a million turkeys, and thousands of people are floating dead in the miles and miles of deep water. By the way, the people are not as a result of the death toll from the storm itself -- that numbered about 40 -- they are from grave sites where the caskets popped up to the surface. Really nasty. Unfortunately for many folks living in eastern North Carolina, the rivers have not even crested yet and the remnants of Tropical Storm Harvey are due to dump another three to five inches of rain there today and tomorrow. Fortunately for us, it is all happening to our east.

In the meantime, I am backed up about 150 emails that really needed to be answered over the past several days, but we have been busy with clean-up. I am going to get to them all today and tonight, catching up completely, then things will get back to normal. So bear with me today and all will be well.


16 September 1999, 11:49 am EDT

The power just went back on and the sun is starting to peak through the breaks in the clouds. I must say that the sky is never so blue as the time just after a hurricane. Personally, I'll take the pollution and haze any day and leave the cleansing by hurricane to someone else.

I am going to leave the generator up for about another three or four hours until I am sure that the power grid in our neighborhood is stable, then I am going to sleep. In other words, don't expect me to answer any emails until significantly later this evening.


16 September 1999, 9:00 am EDT

The real nasty stuff is pretty much over now for us. The rain is down to a medium rainfall (perhaps one-quarter inch per hour) with an occasional burst to heavy (one inch per hour rate,) but not sustained for more than a couple of minutes at a time, and the winds have dropped off to about 20 MPH with an occasional gust to perhaps 35 MPH. The North Carolina Office of Emergency management is already stating that this is the worst flooding ever recorded in North Carolina and the eastern part of the state still has a long way to go. Some areas of the state have received over 25 inches of rain and the heavy rains continue in the northeast part of the state (about 100 miles from us.) We have had about 9 inches of rain where we are with maybe two more inches to come. Floyd is supposed to pull completely out of North Carolina by noon or so, but the rivers will continue to rise for several days. Parts of the Tar River are already 15 feet above flood stage.

Hurricane Floyd is still a strong category one storm moving to the northeast at 25 MPH. CP&L now estimates that over 500,000 homes in eastern North Carolina are without power with 120,000 in Wake County alone. CP&L has over 1,400 workers available to repair power lines and will begin as soon as the wind gusts die down enough so folks working on poles and around trees will be safe.


16 September 1999, 6:10 EDT

That was stupid.

I went out to check the generator fuel gauge (it's still above full) and the sump pump (it's still pumping fine.) The winds were not too bad. So I decided to go to the car and get the big flashlight I left in there in order to see the trees and stuff in the neighborhood.

Our front door faces south. The generator is set on the southwest corner of the building and the door to the crawl space is on the west side of the building. The car, however, is in the parking lot on the east side of the building.

It's amazing how much a building can block the wind. I am not amused. If anyone finds sheets of skin to our southwest, please return them to me. And the flashlight wouldn't even work since it runs off the only type of battery that I don't have -- those big, blocky things.


16 September 1999, 5:34 am EDT

It looks like we got fairly lucky. Floyd has moved in more of a NNE direction after coming in over the Wilmington area. Since Raleigh is due north of Wilmington it means that the center of circulation is moving away from us. We are currently in the northwest sector of the wind circulation with sustained winds of 40 MPH and the highest recorded gust so far at 52 MPH. That is at the airport about 8 miles to our west and away from the storm center.

The storm is now moving NNE with a forward motion of 23 MPH (up from 14 MPH,) and the top sustained winds at the center are down to 105 (from 140.)

We have been listening to the police command center on the scanner and trees started coming down big time about two hours ago. Most of the major roads in Raleigh are impassable and it is unsure how the secondary and residential roads are faring since it is still dark and no one can get to them at this point. All of the streams and creeks are above their banks now and Crabtree Valley Mall (the largest mall between Washington, DC and Atlanta) is about to flood out; the parking lots are already full of water and they have hours of heavy rain to go.

Heavy rain is anticipated to continue through at least 10 am with winds maintaining their current speeds until that time. At this point, the huge problem we have are that the tree roots are loosening up in the wet ground and winds of even less sustained speed than we are currently experiencing are capable of doing severe damage as those trees come down. Remember, oak, elms, and pines don't bend for the most part like palms.

All telco is still up and working fine. We have full dial tone on all the POTS lines and the data lines are running clean with no CRC errors or drops. BellSouth has generators at every POP in the area and the Central Office for the Raleigh area is on high ground.

Kenly seems to be the hardest hit inland town at this point with flooding. The entire town is under water; Kenly is about 30 miles to our east. CP&L is estimating that 270,000 homes are currently without power in eastern North Carolina with much higher totals in South Carolina. It appears that this is going to get much worse before it gets any better since the rivers and creeks will continue to rise for hours or days even after the rain stops. Virginia is next.

I-95 is completely closed between exit 119 and 121 (due east of us by about 50 miles) as the Tar river is completely over its banks.

The generator is still humming and the sump pump is still pumping. We are stilling in very good shape, though it may be days until grid power is restored.


16 September 1999, 2:51 am EDT

Hurricane Floyd is close to making landfall on the lower North Carolina coast. Raleigh is still in the west path of the storm with gusts up to 80 MPH expected through early afternoon on Thursday.

The power went out for good at 2:30 am, the generator went on at 2:31 am. With the combination of backup UPS units and the generator, we have not experienced any outages. We will be pulling an all-nighter monitoring the situation (while we watch TV, eat and work on the computer - so, we have a really BIG generator). We will post further updates here as they occur.


15 September 1999, 6:30 pm EDT

We are finally done with all the set up. The generator is set up and grounded. The windows are completely boarded up. And I am kinda ticked off at Travis, one of our programmers. Some 150 of my muscles are laying around in the front yard and he refuses to go out in the rain and pick them up for me. I guess I'll just have to flop around for the time being.

Right now, we have heavy rain -- after a day of heavy rain -- and the trees are ripe for the winds. It now looks like the eye of the storm is actually going to pass about 40 miles to our east, but that will still leave us with several hours of sustained hurricane-force winds with gusts to perhaps 90 MPH. Not many of the big trees are going to survive.

But we are ready. CP&L is out in full force. There have been trucks rolling all day long. BellSouth is on emergency status and are ready to tackle any problem, even though they don't anticipate any problems in the Raleigh area since everything is underground. We have food which is perhaps the most important aspect of this entire situation.

The bulk of the storm is supposed to start coming in about midnight and last a good 8 to 10 hours.

I'll keep you posted.


15 September 1999

Here we go again...

Not two weeks after a near miss by that pesky Hurricane Dennis (which got us really wet, but without any other consequences,) we are about to enter the Big Time.

Hurricane Floyd is bearing down on North Carolina. The center of the storm is currently located off the Florida coast moving directly toward us at 14 MPH. Current storm projections predict that Floyd will strike land somewhere in northern coastal South Carolina on Wednesday night then move inland Thursday morning. Though the eye of the storm is projected to move to the east of Raleigh, the storm is packing winds of 140 MPH and has a hurricane-force wind shield over 300 miles wide. Tropical storm force winds extend almost 300 miles from the center in all directions.

Current predictions for the Raleigh area include sustained winds in excess of hurricane force with the possibility of winds in the 95 to 115 MPH range if the storm tracks slightly to the west from its projected path. Rainfall is expected to be in excess of 10 inches. Even if the storm moves farther to the east, the impact on Raleigh will be severe and sustained throughout the day Thursday.

The generator is due in this morning. All the windows are boarded up. And all the hard drives are in the process of being backed up as I type.

I apologize in advance if we do not answer the phone today or respond to email in a timely manner, but we're kinda busy. Things will get back to normal by Thursday night -- particularly if I can obtain the services of an orthopedic surgeon to repair the damage I did to myself yesterday preparing for this monster.

And if anyone knows of a good substitute for human skin, please let me know. In that context, I am pleased to announce that the small, razor-sharp, holly bushes buried amongst the lovely bushes outside my windows hve been utterly destroyed.


22 July 1999

BellSouth is working on the F2 circuits today from the crossbox to the doubler. At any give time, we may have only a single T1 line functioning, but we will not lose complete connectivity. The work should be completed by late today. Unless you are doing a massive download via a T1 line or better from your side, you should not see any problems at all. If you are on a T1 and downloading large things, you may see some speed reductions until all lines are back up.


15 July 1999

BellSouth was testing one of the T1 lines last night for about three hours. There was a slight problem which was corrected. No loss of connectivity resulted, however some folks may have experienced stalls of about ten seconds as the line came down then back up and as the load balance readjusted itself.


3 July 1999

IT IS ALIVE!!!!

The offending server is back up and functioning flawlessly. We are happy, though tired. It turned out that there were three problems that happened at the same time. One, a SCSI cable went south. Two, the FTP software corrupted. And three, we somehow got corruption of the primary drive directory structure such that some folders were actually on there twice, but only appeared once in the file list. Needless to say, any one of those would have given us fits, but all three together resulted in thoughts of homicide dancing in our heads.

The up side is that the only time the machine stopped serving was during reboots and only FTP was directly affected during the move. Also some folks had problems seeing their sites not because of problems on this side, but because their ISPs are too ignorant or cheap to abide by my DNS time-to-live. You see, I have all DNS entries set to expire in one hour so that if I move a domain from one machine to another, any information reflecting the old IP number is supposed to die at ISPs in one hour then be refreshed the next time someone on that ISP requests the site. Most ISPs abide by that rule, but some are just plain ignorant and cache the dead information for one to three days. They do not play by the rules and their clients suffer as a result. The fix is two-fold. First, call up your ISP and tell them to flush their DNS cache; that solves the immediate problem. Then chew them out for not following the standards of recognizing TTL parameters.


2 July 1999 - Late

It's now about midnight and the server is completely reconfigured with all new software. I am going to take a break and grab dinner, then start reconnecting all the FTP accounts. Each one needs to be manually configured and there are about 400 of them. Each one takes a bit less than a minute, so you can pretty well figure out what I am doing till the sun comes up. All should be completely back to normal by morning.


2 July 1999

That did not go very well at all. We have suffered failure of the FTP software on one server that handles some 400 domains. It is serving web pages just fine, but fixing the ftp server will require some downtime of the server. Therefore we will be monitoring the machine all day and will address the software failure later this evening when the machine traffic slows down. We're aiming for no more than 15 minutes of server downtime in the middle of the night (tonight). Thank you for your patience while we solve this issue.


1 July 1999

The servers are fine, but ftp is temporarily turned off. We are upgrading some of the hard drives on our servers and doing so seamlessly so that your web site keeps serving. Since we are moving files, we chose to shut down ftp so that we do not overwrite any new files that you might put on the server during our upgrade. We estimate that the ftp for all accounts will be restarted by 6 AM 2 July 1999, although your ftp may be available earlier as we restart each one by one.


26 April 1999

Had a strange outage that only affected four clients, but it was weird nonetheless. We had a 12-volt power block plugged into the UPS unit that served four colocation machines. That power block was barely resting on the top part of the master on/off switch. Over the past several months, the vibrations of the machines sitting on the same shelf as the UPS unit caused that power block to settle into the switch and turn it off. Needless to say, when you shut off power to the computers, then tend to turn themselves off.


16-20 April 1999

We will be testing our new automated rebooting system this weekend. At times, we will be intentionally crashing the servers to check recovery and remote rebooting times. In early trials, we are able to automatically reboot any crashed server in less than two minutes. We are happy. And since we can now fully reboot any hung machine from anywhere, we can actually do things like normal people do such as eat dinner out. We are still happy.


8 April 1999

We took a five minute telco bounce for no apparent reason right at midnight. Given that the time was so precisely at the top of the hour, I thought it was BellSouth doing some routine maintenence and forgetting to tell us. Rather, someone with a large butt apparently swung too wide at the NOC and knocked something loose. The appropriate people have been yelled at.


16 February 1999

For the past week and for the next week or so, the various machines will be bouning up and down for very brief periods of time every couple of days. The typical outage will last about three minutes. We are in the process of loading the new version of MGI on all the servers and though we will try to confine our configuration to the middle of the night, sometimes things need to be done in the day time. We will keep any disruption to a minimum.


4 February 1999

The phone's been ringing off the hook today with folks complaining that they can not reach their web sites, FTP to their home space, or check email. There is nothing wrong with our system nor is there anything wrong with BellSouth. There ARE major problems affecting AOL, Mindspring, SWBell, and a handful of other providers. Therre may also be problems at the major peering point, but I can not effectively verify that. Today has apparently been a rough day on the Internet and the only thing you can do is ride it out. Perhaps it will be better by late tonight.


22 January 1999

Around 5:45 this morning, some genius in BellSouth Provisioning decided to install a new circuit without telling anyone (there or here), taking down most of Raleigh in the process. After about a one hour outage, we are taking stable traffic again. The responsible party shall be duly flogged with a wet noodle!


14 January 1999

Whoops...

As you may have noticed, FTP services are down. I did a major reconfiguration of the web server this morning and every alias for every client broke. fortunately, nothing happened to the web serving capabilities of any web site, but FTP is down hard. I have to reset each alias by hand -- one by one -- to bring them back up. That should take at least ten hours (like I didn't have anything else to do today.)

If you are trying to FTP in and need to change files before I get to your specific domain, call me at 919-852-5262 and I will skip to yours and hook it back right then. And don't be shy to call. That's what I'm here for.


10 January 1999

Now THAT'S the way things are supposed to work. Some idiot BellSouth contractor, while picking his nose by the neighborhood crossbox, decided to mess with one of our T1 lines. Of course, it went down for five minutes. But the load sharing software worked perfectly and the other lines picked up the slack without even batting an eye, something which can not be said about the BellSouth contractor who is presently trying to pull a 2X4 out of his pupil. How the 2X4 got there is something I probably do not want to get into at this moment since there is still 6 years, 364 days, 23 hours, and 52 minutes before the statute of limitations runs out. I will, however, entertain a medical discussion on the merits of various methods used to dislodge a cinder block suppository, such a discussion being appropriate to the above-mentioned BellSouth contractor as well.


9 January 1999

The web server crashed and was automatically rebooted today. Total downtime was a bit under three minutes. Since I was in Florida basking in 80 degree weather, I am not exactly sure why it crashed, but it didn't happen again, there was no need to get Chad or Travis on the problem, and it has been stable since. It was probably hungry.


3 January 1999

We had a glitch of the stupid graphic counter software today that interrupted service to all graphic counters for who knows how long. I really hate that thing. Fortunately, we are within two weeks of replacing it entirely with something we wrote and know works properly. When that is done. I am going to take the existing graphic counter program, put in on a floppy disk, and toss it into the fireplace.


2 January 1999

We did some work on the secure server last night and had the thing bounding up and down for about six hours during the early morning hours. It was never down for more than about ten minutes at a time. I need to finish up with the software install at some point, but I will wait until the weekend to do so.


1 January 1999

Nothing to report. I just didn't want this space to be blank. :)

Return to the Server Status Menu

 


       Home | Services | Order | Resellers | Affiliates | Server Status
       Tutorials | FAQs | Online Payments | About Us | Contact Us

Copyright © 2024 PagePlop.com Comments.