Xfernal's Techno-Dev Mumbling

Been so busy lately, I have already started neglecting this blog. But, one of the reason I have been so busy is because we have been preparing for our new server. We recently caught a great deal on a Dell Poweredge 7150. This is a powerful server. Now I know that the latest Itanium processors perform better than this first model, but you just can't turn down an opportunity to buy world class enterprise hardware for pennies on the dollar. Afterall, my XML search project was in dire need of some serious database processing power.

Here are the basic specs:

· Quad Intel Itanium 800Mhz CPUs

· 12Gb PC100 DIMMs (Upgradeable to 64gb)

· 4 x 36Gb SCSI Hot Swap Hard Drives Ultra160 10k

· Perc3/QC Ultra 160 Raid Controller w/128mb Memory + Battery Backup

· Eight 66Mhz hot-pluggable expansion slots

· Quad 700 watt Power Supplies

· Integrated Ultra3 Dual Channel SCSI Controller

· LS120 Drive

· CDrom Drive

· Dual Integrated NICs

· Dual USB Port


The whole 64bit architecture is definitely new to me, but I am hoping everything runs as it is suppose to with Windows 2003 Enterprise Server and SQL Server 2000/2005 (have not decided which to install yet).

Here are some quotes from a press release about the 7150:
“The Itanium processor and its 64-bit architecture extend the capabilities of Intel servers by exponentially increasing the amount of memory that the microprocessor can interact with, meaning the system is able to process more information at a faster rate than ever before. Additionally, the new processor enhances floating-point performance, the system's ability to perform complex calculations, for faster execution in scientific applications, such as in engineering programs used for modeling and testing. The new processor incorporates error-correcting technology, known as machine check architecture, which enhances the system's ability to compensate for processing errors automatically and ensures greater levels of availability and data integrity.”

“Nasdaq is testing a prototype Dell PowerEdge 7150 system for two classes of mission critical applications. The first is a class of memory intensive applications used to replicate and distribute stock market data on an internal data "pipeline." The second class of applications deploy Microsoft SQL Server in support of Nasdaq's ever growing Internet and intranet presence. According to Gregor Bailar, executive vice president and chief information officer, Nasdaq, "With daily transaction volumes on our network peaking at more than three billion shares per day, we are looking for systems that will support those transaction volumes and our need to access huge amounts of data dynamically. We expect the Itanium-based systems to allow us to grow our capacity and be responsive to our customers, while lowering our cost of ownership and administration."”
Source: Dell Readies Itanium-Based PowerEdge Server

Yes, this is older technology, but it sure beats shelling out at least 5 figures for a new server. Brand new, this server cost about $66,000 and we picked it up for less than $2,000. Best of all, this server is still under Dell Gold Support, and leaves us with plenty of extra room to grow!

What does this buy us? Horsepower! This monster server should handle an insane number of sql queries. And if it slows, I can always add more RAM (up to 64 Gb). The crawlers/spiders I have been working on utilize a central database for managing the queue and index. This simplfies the complexity of multiple crawlers on different machines. By storing the data in the central store, a "smart spider" can be achieved by dynamically altering the queue based on retrieved data and keywords. By emphasizing importance of "links" based on the content and keyword depth of a retrieved document, more important documents are retrieved first, before less important documents. The importance of a document can be had at the document level as well as the domain level. As important documents are accumulated from a domain, the domains "ranking" is increased, which affects the document level ranking. We can also alter ranking based on domain/site performance. If a domain typically has a slow response, generic content, or large number of broken links we can queue any page lower, allowing the spider to retrieve from high performance sites first to get the best and latest data. This speeds the spider process by getting the info from "important domains" first. Why waste time with sites that do not behave as nice or are not updated as often? Similar algorithms have to be in use by the major search engines.

This new server has not come without complications in our environment. We had to install a new 20 amp power outlet for this big beast. With 4 - 700 watt power supplies, this pushed us over the edge with our current use of 35 amps. 55 amps should be sufficient for a while...

We also had to buy another APC UPS unit (2200 rack mount) to handle the power backup. And a power conditioner for the new 20 amp line, in case of generator use. Once again, this caused another server shuffle in the rack. We decommisioned an old HP NetServer to free up 8U of space, only to fill it with the 7U PowerEdge.

Also, this server has 14 fans! Two fans for each power supply, 4 huge 6 or 8 inch fans and 2 - 4 inch fans. We have not turned this beast on, but it should sound like it is on the runway preparing for takeoff when we finally do.

So, I will remain very busy in the next few weeks as we get everything in place for this to become a production server. Lots of testing will be required and with such little time available for such luxuries as blog writing, I am afraid this will be my last post for a bit. Until then....