Hosting upgrade complete

Well, it’s done. Everything was moved smoothly onto the new server. A few users may need to reset their password; outside of that it should have been seamless.

I’d elaborate on the details – but who cares? More space for everybody!

Hosting upgrade on the way

Demand for Web hosting is on the rise. Our new WordPress self-manageable sites are a hit. And an upcoming partnership with KBI to provide free Web hosting for area small businesses is on the horizon. Take it all in, and suddenly that shiny new Web server we moved into three years ago is looking a little undersized.

So it is that we’ve begun the latest in our long string of Web server upgrades. But unlike last time, this move isn’t being brought on by any calamity. On this go-round, we have the time and flexibility to make it a smooth transition for all involved. Chances are you won’t even notice the change. (Unless you’re a WordPress user – in which case you’ll likely enjoy a larger disk space quota.)

If you have any questions, contact us. If we have any concerns about moving your site, we’ll contact you. And as the next few weeks pass, we look forward to seeing you on the new server.

End of an era: raqpaq and scooby halted

The above picture was what you’d see if you entered into your web browser’s address bar. That is, until 11pm last night.

It was at that time that our two remaining servers at NCIS – raqpaq, our former shared hosting server;, the last remaining piece of the once-glorious Kidsnet system – were officially turned off for good.

This marks the end of an era… one which started nearly a decade ago with a humble Linux box serving pages for the Lone Pine community center, and which grew to consume a whole corner of the NCIS “datacenter” at its peak some five years ago.

Times change, though. NCIS isn’t what it used to be, and neither are we. Our needs have changed, and our customers’ expectations aren’t the same as they were a decade ago. Now sites are bigger, more dynamic, and relied on moreso than ever. That’s why we put dala online… and that’s why we’ll probably keep expanding in the years to come.

scooby and the console, in its former home

But for today, give a nod to the tired iron that gave dozens of area establishments their Web presence for so many years. It’ll be torn apart and hauled back to our office next week for a much-deserved retirement on a shelf downstairs. Or perhaps to be reused again someday…

The new webserver: IT’S ALIVE!

Well, the new server (whose name is dala, by the way) is now up and running.

In fact, the page you’re seeing right now was served by it.

So far the transition has been fairly smooth. DNS changes propagated to all the major nameservers in a matter of two or three hours – lightning fast as compared to the usual 12-24 hours. Our password database was abandoned and the users were all moved by hand, but everything else was scripted and “automagically” took care of itself.

About the only config issue I ran into was with our MTA. I’ve always disliked Sendmail, mostly due to my lack of understanding it (but then, who can really claim to understand Sendmail’s arcane config-file format?). Even so, I could never quite bring myself to make the move to something more modern – “if it ain’t broke, don’t fix it”, right?

However, after spending a few hours struggling (and failing) to make Sendmail play nice with saslauthd on the new server, I finally threw in the towel. Now we run Postfix… and yes, it really *is* much nicer.

The last thing to be moved is the Mailman mailing lists and their associated archives, which is happening right now. So far so good!

Will this make all the customers happy? Tomorrow should give me a better idea of that… but for tonight, there’s still lots more non-server-related work to be done.

By the way: the admin interface is now at If your password isn’t working, call and we’ll be happy to reset it for you.

And if you happen to be missing anything, don’t worry – the old server (raqpaq) is still running, and we can easily log into it for you and retrieve anything that might have been overlooked. Or you can dig around yourself… just telnet, FTP, SSH, or what-have-you to and take your last look around. (We’ll be taking it down for the last time sometime later this week.)

Web server move officially begins 7/8

When NCIS’s T-1s went away a few months ago, we ended up with severely reduced bandwidth – 512Kbps shared with three other customers, as compared to the 3Mbps we previously enjoyed (and practically had to ourselves).

After many attempts to get moved onto a different connection here in Mora were fruitless, I finally gave up a few weeks ago and called another ISP. Since then I have set up a new server in their facility, where it will be on a fully redundant 100Mbps connection that is a only two hops from the Internet backbone (even better than the T-1s!). This will mean much better throughput, no interruptions in service, and greatly improved response times.

Pretty much every issue we’ve had in the past months (downtime, dropped connections, slow connections, SMTP issues, etc) can be attributed to the poor connection we’ve been on. We used to only have such issues on the rarest of occasions. Our new ISP assures me that we will never have such issues with them… and from what I’ve seen, I tend to believe it.

The first sites to be moved will be Four Corners and the remaining assets of Kidsnet. Once those are done, it’ll be the Kanabec Systems sites and services, then 332 Media (including their all-important scripts that generate weather and news updates for KCIZ-AM/FM/TV). And once that’s done, we’ll start in on the paying customers – one at a time – until everyone has been moved. This will probably take several days to complete, so don’t worry if you don’t notice any differences in your hosting right away.

When putting together the new server, we were able to make some changes we’d long wanted to implement, as well as switching some of the software involved. You may notice that the Cobalt management interface has been replaced by Virtualmin, a more modern package that accomplishes the same purposes.

Most of the new server’s facilities will work the same as the old one’s – but if we notice that the differences might impact your site or operations, we’ll be sure to call you and work through any potential changes ahead of time.

Also, be aware that we will be making some DNS changes as part of this move, so there may be random brief periods of downtime while changes propagate. Unfortunately there is nothing we can do to prevent this – but be assured that everything we do have control over will be moved in the most graceful manner possible.

Again, thanks to all our existing customers for their patience, and we look forward to seeing your site over at the new server!

Doing the “Hurry Up & Wait”

Well, several weeks have passed since the NCIS/Onvoy incident unfolded. The original plan was to get our equipment off the temporary connection within a couple of days, and move it to something more permanent. But (as you probably guessed by this point), that has yet to happen. NCIS is still working on obtaining additional bandwidth for us – but it’s taking a lot longer than we hoped.

The good news is that we’ve got our temporary connection as groomed and tuned as it can be. Packet loss is at a minimum, and we’ve gotten a handle on the congestion situation. But the bad news is that it’s still only a 768K pipe… half of what we used to be on.

So we wait. The word is that we’ll be back at full capacity in another week or so, but it’s hard to say. Until then, keep on keepin’ on, and thanks to our hosting customers on raqpaq for being good sports throughout this process.

Outgoing mail bouncing?

Many customers have reported that some (but not all) of their outgoing mail is being rejected by the recipient’s site. This post addresses that problem.

Some mailservers, especially those at larger organizations, have spam filters that utilize a variety of tactics to spot potentially forged mail. One of these tactics is to compare the sending mailserver’s name to it’s reverse DNS entry. If the two do not match, it is assumed that the sending mailserver is lying about its name (it may have been set up for the purpose of sending forgeries, or for trying to conceal a spammer’s identity and thus shield them from any fallout). In any case, the filter will reject mail under such circumstances.

Since moving to its new address on the temporary connection (see earlier posts for details on what happened), our mailserver’s reverse DNS entry has not been updated. Though we are in the process of getting it changed, it will likely be a few days before anything takes effect. Until that time our mailserver’s forward and reverse records will not match… and so filters using the test outlined above will reject any mail sent through it.

If you are having problems with mail not being delivered due to this, you have two options: either wait for the changes to take effect, or use another outgoing mailserver.

Your ISP may operate an outgoing mailserver. If so, using theirs would probably be your best option. Some common ISP’s outgoing mailservers include:

WildBlue: (port 465, SSL required)
CenturyTel: (authentication required)

Just reconfigure your mail client (ThunderbirdOutlook ExpressApple Mail, or what-have-you) as appropriate, and you should be good to go.

Failing that, we have permission to provide our customers in need with access to another friendly local provider’s outgoing mailserver. If you need details, call us – we’d be happy to walk you through it.

And, of course, we’ll post here once the reverse DNS situation is resolved.

UPDATE: 2/3/10 10:00a The reverse DNS settings for NCIS’s outgoing mailserver seem to have taken effect sooner than ours. As such, we’ve changed our configuration – now any mail sent through our outgoing server will be relayed through their outgoing mailserver before continuing on to its destination. This should cure the problem until the reverse DNS situation is over and done with. If you have any issues, let us know.

Onvoy-NCIS saga ends

Well, the Onvoy saga is over, but the fun is just beginning: it seems that what began as an outage has turned into a parting of ways between NCIS and Onvoy.

Contract negotiations between the two companies were going on before the outage occurred, with NCIS already on the verge of walking away and finding another bandwidth provider. But being without service for three days was the last straw, and so as of today NCIS is on the hunt for a new ISP.

[ Side note: This doesn't come as a surprise - or a disappointment - to me. Onvoy has changed hands several times over the years, and the latest incarnation seemed to be the lousiest yet. 'Smatterafact, the only time I can ever recall seeing Gary (owner of NCIS) mad was after dealing with an Onvoy tech a couple of years ago... and that says a lot. -KT ]

What does this mean to us? Well, it means that our servers will be on the temporary connection (mentioned earlier) for at least the rest of the week while NCIS gets a new provider. It also means that our next move isn’t as definite as before – we had planned on getting back onto NCIS’s Onvoy-fed connections over this coming weekend, but now we’re not 100% sure just what (or where!) we’ll be moving to. This will at least mean another long weekend of address-swapping and DNS nightmares, and might even mean that our servers will be physically moved to another building, or perhaps another town (we’re working on several possible plans).

But, what does all this mean to you? Probably not much – just that we’ll be down at various points throughout the coming weekend, and that there may be some new numbers to enter into your routers and other equipment yet again next week.

At any rate, rest assured that we don’t intend to let this put an end to our hosting or other services. It’s just another bump in the long road we’ve travelled – and will continue to travel – in our quest to bring you the best services we possibly can. Hang in there!

Onvoy outage

At around noon on Friday, January 29 2010, we experienced an Internet outage that cut off access to our primary Web and mail server.

It seems that the problem was two hops detached from us – our systems were OK, our ISP (NCIS)’s systems were OK, but their ISP (Onvoy)’s systems were not. This caused a trickle-down effect that cut off all of NCIS’s T1-fed customers… such as us. It also resulted in all of NCIS’s servers (mail, web, DNS, etc.) being disconnected from the Internet.

In the 16 years that NCIS and Onvoy have been doing business, a prolonged outage such as this has only occurred once. In our 10 years of working with NCIS, it has never happened – until now.

We installed a temporary connection over the weekend, provided to us by the good folks at YouBetNet. This connection is nowhere as nice as the one we normally use, but it will keep things going until Onvoy gets their act together.

Mail sent to our customers on Friday afternoon and over the weekend started arriving on Sunday morning, and will continue to arrive throughout the day today as other mailservers throughout the Internet discover our new routing and attempt to re-send the undelivered mail.

Any customers experiencing connectivity issues at their homes or offices should confirm that they are notusing a DNS server whose address is 206.146.216.x – if this is the case, you will need to change the address to something else. We recommend (the Google public nameserver) as an alternative.

If you are having any hosting-related issues as a result of this situation, we encourage you to call us – that way we can figure out what’s going on and get it fixed.

We thank all our customers for their patience, and look forward to serving you.