I’ve decided to start keeping a journal of all of the various business crisis that we generally have to deal with on a somewhat regular basis here at Wandertec, Inc. My primary objective is to give myself a record to refer back to when a similar problem occurs. But I figure that my trials and tribulations might also provide my legions of readers with a few insights into running a web based bicycle business.
The majority of our business activity is through our websites Bike Trailer Shop.com and Bike Bag Shop.com. Whenever there are problems with the websites, we go into full on crisis mode. These problems revolve around issues such as our servers having problems, our payments not processing, our email not working or the code on our site not working properly.
The most common problems for us have been server problems. Lately our site has been shutting down due to some form of server overload. To troubleshoot it, all we can really due is theorize what the problem is, apply a fix and wait to see if it happens again.
We moved our 2 web shops to a new server last summer. The move overall seemed to be effective in eliminating the slow performance and frequent shut downs that the Bike Trailer Shop had been having on our old servers at APlus.
Our sites are now on a Virtual Private Server (VPS) hosted at Network Solutions. In the last few months, we started experiencing the site shutting down. This was easily fixed by resetting the server, however we knew that this shouldn’t be happening. With the frequency of the shutdowns increasing we started looking more deeply to see if our sites file structure could possibly overloading the server. We restructured some of the filing system, however the problem persisted.
The latest diagnosis was made by Jeff here. He had noticed that the name servers for both of our web shops were located on the same server as the site files themselves. Jeff assessed that this would possibly overload the system with self referring loops. Additionally, this goes against common practice of putting name servers and website files on separate servers so they do not tax the same server resources simultaneously.
With this theory, we decide to move the name servers to a separate Network Solutions server. To do this, we have to 1. Move the name servers 2. Point the domain names to the new name servers 3. Setup the records on the new name servers to properly access site files (A records) and email (MX records).
This process is difficult enough to figure out and initiate. It is made even more difficult in that the name servers take awhile to propagate across the internet. While propagation is happening, some people will be seeing what the old name servers want them to see and some will be seeing what the new name servers want them to see. If the reason for changing the location of the name servers is to try to resolve any issues, it is difficult to know when the new name servers have propagated, though there are network tools to see if where you are being pointed.
In our transition to the new name servers, we encountered another issue. Network Solutions, does not allow for the MX records to be easily adjusted unless your domain is hosted with them. With our domain on another domain registrar, we did not understand why we could not adjust our MX records. In the process of attempting to adjust our MX records, we inadvertently disabled our websites from communicating with other servers. Doing this disabled our PayPal payment processing, USPS rate calculation and FTP access. Our email was also disabled because we had not effectively loaded the MX records.
We were under the impression that the PayPal, USPS and FTP issues were related to the MX records not yet propagating. After we waited a day and nothing was working, we contacted Network Solutions. We soon learned that we had not only not edited the MX records but we had additionally disabled our server’s outbound communications function. Network Solutions was able to immediately reset the servers and reestablish our PayPal, USPS and FTP access. It turned out that with Gmail as our email client, our email had some default forwarding (to my personal email address) set up. I got a flood of 2 days of our business emails that I thought had been lost.
As far as the properly setting up the MX records, we have 3 options:
1. Have Network Solution manually edit our MX records (we would have to wait until the weekend was over)
2. Move our domain registration to Network Solutions so that we could access there dns tools.
3. Point the domain at a new Name Server (not on Network Solutions) that includes dns tools, so we can easily adjust the MX records in the future.
I ended up choosing the first option. Though it requires us waiting for Network Solutions, it won’t necessarily take that long and we don’t anticipate any need to adjust the MX records very frequently. The second option, may be something we do in the long run if we need to adjust the MX records more frequently, but we didn’t want to have to wait for this time consuming process to go through before we could make update our MX records. The third option would be yet another name server switch. I did not want to go through this again with all the problems that seem to surface every time it is done. However, for our future sites on Network Solutions, we will likely setup the name server at our domain registrant so that we can have access to the dns tools and not be reliant on Network Solutions manually editing changes to MX records.
So in summary, in attempting to solve one problem we unleashed a bunch of others. We are now keeping our fingers crossed that putting the name servers on a different server than the site files was worth while and that the website stops shutting down.
Some of the lessons learned:
1: Do not do major name server or server switches on a Friday
2: Talk to our server host technical support before making these types of switches.
3: When something like this happens write about it so I can refer back next time around. (done)