Making Your Website Visible

Making Your Website Visible

Introduction

As the Web continues to expand preparing your web site in such a way that it becomes visible to the audience you are trying to reach requires some planning, and continued work after the site is created. There are various levels of visibility that can be obtained, and as with many things, the more visible you want your site to be the more time and effort is required.

Many people think of site visibility as being directly related to how high your site appears when someone goes to a search engine and enters in some keywords that pertain to the topic of your site. While it can be quite nice if your site does "sift" towards the top, in some cases this may never happen, or it may not happen reliably. To start, there are more and more search engines being used on the Web, and most have different methods for categorizing sites and different sets of Web page data. Here is a quick rundown of some of the more popular search sites, or information portals, and information specific to each:

Yahoo (http://www.yahoo.com) - Arguably the most popular site on the Web. What many users do not realize is that Yahoo does not actively index Web pages, but relies upon users to go to the site and self-register their sites under the appropriate category.

Excite (http://www.excite.com) - The next most popular site on the Web, but the most popular search engine. Excite claims to have 30% to 40% of Web pages indexed on their site.

Altavista (http://www.altavista.com) - The original big search engine. Originally owned by Digital Equipment Corporation. Now owned by Compaq. Altavista pioneered the use of fast search algorithms on tremendous amounts of data for indices of Web pages. They also pioneered many of the algorithms used for storing Web pages, deciding what keywords apply to a page, the use of metatags, and the concept of "robots" or "spiders" that catalogue Web pages.

America OnLine (http://www.aol.com) - Since AOL is the largest Internet Service Provider (ISP) in the world, they are also one of the most visited portals on the Web.

All The Web, and All The Sites- (http://www.alltheweb.com, and http://www.allthesites.com) - A new site with the goal of returning precise results to detailed queries, and of indexing every page on the Web.

Ask Jeeves (http://www.askjeeves.com) - This site has the user ask questions to find answers.

There are many, many more search engines and portal sites. A few include Northern Lights (http://www.northernlight.com), Go, by Disney at (http://www.go.com), Netscape's Netcenter (http://home.netscape.com), HotBot (http://www.hotbot.com) run by Lycos, Google (http://www.google.com), and Internet Explorer's default home page (http://home.msn.com).

One nice summary of some search sites is available at http://library.uoregon.edu/network/srchweb.html.

A number of these sites simply buy service from companies like Altavista, use their databases and search engines, but emphasize information in different ways. Others have their method for finding new and current sites on the Web, indexing the data, and searching this data. Some sites index pages most strongly based on the page title, others pay strong attention to the meta tags in the page, and others pay attention to the number of times a word appears in the page. Most search engines check to see how many other sites link to yours to help determine how high up your site appears in search results.

Many sites have used the trick of adding text the same color as the background that includes words that they think people will search on to find their site. At first this method generally worked well as these sites would often float up high in many search engines, but lately most of the major search engines have come up with algorithms for detecting this trick and penalize such pages by dropping them down lower in search results.

Planning Ahead

Before continuing there are some important things to think about. These include:

How are you planning to get people to your site? This is the number one question. By search engine results, from other sites, by sharing links with others, by using traditional advertising such as newspaper, television, radio, by word of mouth, banner ads, or by other methods? Some combination of these? You'll want to think about this as you go forth with your site design
Will your site be static, or change a lot? Most search engines will drop your site's visibility if it does not change every 3 to 6 months.
How is your site created? Are you using a database driven site? Is it text based HTML pages? Etc.
Are you going to use frames? If so, read the section "Frames and Dynamic Pages" for hints on how to deal with frames and visibility.
Are you creating a site that is part of a popular theme? For instance, if your site has information about computers on it, or the Internet, then getting it to appear higher up in search results may be very difficult, to basically impossible. In this case you may want to consider other methods for getting people to your site, depending on your budget. Generally most folks don't look past the first page of search results, which is usually 10 to 20 items at most. Considering the several hundred million Web pages already available (March 2000) popular topics are likely to contain hundreds of thousands of pages that pertain to them.
If, by chance, you have pages that you don't want to appear in search engines, then you can include a file called "norobots.txt" in the directory where these files reside. Not all search engines abide by this convention, but many do.
Finally, many search engines only pay attention to the main page on your site (index.html, index.htm, default.html, default.asp are common files), or pages in the root level directory of the site, or (and this is most common) pages that are directly linked off your main page. Others pay attention to the main page in each directory. Others will go 1 or 2 levels deep and then no more. And, finally, a site like "All the Web" (http://www.alltheweb.com) is aiming to categorize everything.

Improving Your Site's Visibility

OK, with all this in mind it's time to figure out how to make your site as "visible" as possible. Here are the basic steps to doing this, not necessarily in order of importance:

Use descriptive titles on every page on your site.
Use Meta tags. These are very useful as the description of the page, if found, is often whatever you place in the "description" Meta tag. The "keyword" Meta tag will often help your page, or site, to sift up higher in search engines based on the keywords you place there, but not all sites use this tag.
Keep accurate logs of visitors to your site and each page. The most important piece of information that you can get is what keywords were used to find your page. Many search engines will pass this information to your site if someone clicks to a page on your site directly from the search engine. Based on these keywords you can then refine the words you stress on your site, or use in your "keywords" metatag to make finding your site easier. It's often hard to guess what people will search for when finding your site.
Register your site with as many search engines as you can as this may speed up the time it takes them to index your site. Note that if you do nothing you can expect to wait up to 6 months before your site will appear in most search engine's database. But, this period is random, and you may appear sooner, or later. Note that even if you register it can still take quite some time before your site is indexed.
Register your site with information portals like Yahoo (probably them most important one). Deciding on your site's category can often be difficult if you have multiple areas of interest represented on your site.
Get your site linked on as many reciprocal link pages as possible. Note that many companies create nothing but pages of links that link to their customer's pages just to increase site visibility. These pages are more effective if they sit on unique domain names in the top level directory of the domain. Some sites, such as Google, use this criteria strongly for determining how high up in search results your pages will be.
Once you have a good feeling for the keywords that work for your site, then include these keywords in your site's text as much as possible.
And, with the same keywords that work well try creating a page, or pages, that is linked off your main page that contains a bunch of text describing your organization, and that uses the keywords that you like. Make the link off your main page a hidden link (like a single pixel image, or a "." the color of your background). Then, if possible, set things up so that if someone goes to this page (usually because they find it in a search engine) that they are automatically redirected to your site's main page. Under UNIX you can do this using ".htaccess" files, or creating a symbolic link.

Notice that adjusting the keywords on your site to correspond to what people use to find your site requires the keeping of logs, or the capturing of this information in your own pages and writing them to your own log files if your ISP does not keep logs with sufficient detail. To view what the most important keywords were you will need to use some Log analyzer software. For UNIX there are many free ones. For all platforms you can purchase something like WebTrends (http://www.webtrends.com), which will give you a list of keywords used to find your site or certain pages if this information is in the logs it is reading. Another product that appears to work well when you are using virtual hosts is WebSuxess at http://www.websuxess.net/english/index.html. For a more complete list see Yahoo's list of Log Analysis tools.

A number of sites run programs that "touch" their pages on a regular basis to make them appear updated all the time. This can help to keep your site "fresh" and listed higher in many search engines.

This article makes no mention of what's involved if you want to generate visitors using advertising techniques. These can be very successful. The one piece of advice is that if your advertising campaign is successful be sure that you can deal with whatever potential number of hits you might generate.

Frames and Dynamic Pages

To start, let's deal with frames. The main issue here is that (for example) a file like index.html is not the actual file that contains the page information for your site's main page if you are using frames. There are many ways to use frames, but a very typical method is to have a left, top, or bottom navigation area that does not scroll while in a separate frame you have actual content. This means that index.html points to one frame that has the navigational elements and another frame with the actual content. When a search site checks on your site and looks at index.html they will, often, completely miss the contents of the other two files referenced by index.html. This means that search results may be poor or incomplete. Naturally this is just one example of many possible situations.

OK, so how do you deal with this? The easiest method is to be sure that you have the full content of the files referenced to in the main frame file (in this case index.html) in a section referenced by the <noframes> and then </noframes> tags. In addition, be sure that you have any of the meta tags needed included in the primary frames file as well. Finally, don't forget to include the correct title for the page in this file as well.

You may have already noted a major issue with the statements above. If you need to use the <noframes> </noframes> construct, then you are essentially doubling your work. This means that each time you make a change to your file you must also make the equivalent change to the <noframes> </noframes> section in the primary frames file. In addition, you will probably have to spend some time designing this section to recreate your "framed" version of the page in a reasonable fashion. As an aside, there are some browsers that cannot view frames, so having this section will insure that people using these browsers can view your site. Generally the number of people visiting your site who cannot view frames will be very low.

What about dynamically created pages? Generally these are pages that are either partially or totally generated from a database. This means that the actual file sitting on your server may have only a few lines of HTML code, or simply be a reference to a database entry using something like Perl or Visual Basic Script (ASP pages). In general, if the page being generated appears to contain all the necessary HTML code to create a search engine reference, then this should not be a major issue. The reason for this is that the Web server is what returns the page that is generated for most search engines to see. As long as this appears as a reasonably formatted page then search engines should be able to use it to create references in their database.

Counters and Logs

Finally, you may have noticed many sites that have counters on their main page, or even on each page. While this gives some indication of traffic, it is not a very effective method. Many counters count every time the page is loaded, and not each time a unique visitor sees the page. Generally logs kept by your ISP contain much more useful data as you can see both raw number of hits to your site, to each page, and the number of unique visitors to your site and each page. You can also see popular times, who's visiting (although AOL skews this dramatically as all AOL visitors appear to come from Virginia), and what pages receive the most visits. A counter may slow down the loading of your page, can increase system overhead dramatically if your page becomes popular, and is generally not very useful in terms of information presented.

In some cases, implementing all of these methods will require that you have complete control over your site as not all ISPs can offer the specialized services necessary to make your individual site as visible as necessary. The largest issue is often what logs are kept, and in what format. The tradeoff with this is that maintaining your own hardware can be extremely expensive and time consuming, particularly if you require that your site stay up. Your best bet is to carefully research hosting provider and ISP's to see if they can offer you the specific services you need for the site you are building. Yahoo offers a list of services that compare various Web hosting companies available today.

Conclusion

Making your site visible on the Web can be a time consuming, labor intensive process. But, taking a few basic steps, and avoiding a few pitfalls, can help to improve your site's visibility dramatically. Most of the information in this document will change over time, so if you see errors, or have comments we would love to hear them. Feel free to send them directly to the Network Startup Resource Center at nsrc@nsrc.org.