What I want to talk about isn’t as complicated as it sounds. Basically, in collecting daily site specific metrics for the purposes of SEO, you sometimes get some bad data from your third-party sources.

These bad data points can skew your graphs and make it almost impossible to visually derive any useful information from them.

For example:

The graph above (click to expand) shows the number of pages for a 6 month date range. In early January the data spikes dramatically upwards. This is clearly an anomaly, as the day after, the data is back in its normal range. The trouble here is that this anomalous point skews the entire graph, making it impossible to derive any real insights from it. The above graph is rendered useless by that one piece of anomalous data.

Here is the same graph WITHOUT that one bad piece of data (click to expand)

It’s amazing how one bad data point can skew your entire graph, isn’t it? Now we have an image that is useful!

Here’s some (hackish & sketchy) code to identify these statistical anomalies and replace them:

Alot of people recommend MaxMind or some other geoIP database, but really, thats a pain in the ass…there is a much easier way to do it.

On your landing page, just include the following in header.

<script type=”text/javascript” src=”http://www.google.com/jsapi”></script>

This script will calculate your visitors location for you….and all you have to do is pull the ClientLocation variable!
For example, if I wanted to display a popup with the users city:

alert(google.loader.ClientLocation['address']['city'])

Seriously, its THAT SIMPLE.

Have fun!

I’ve recently undertaken a personal project that I hope will make my day-to-day activities on a the web a bit more efficient. I’ve always dreamed of having the perfect home page for myself; one that had all my most crucial data at a glance. Unfortunately, all the services I’ve tried have fallen short (*cough* netvibes *cough*). So, I’ve decided that enough is enough; I am going to build my own dashboard.

The dashboard will take quite a few cues from the interface of The STAT Codex, in so far that it will be window/panel based and entirely modular, but it will be customized to my needs.

At the very least, I want the dashboard to pull the following data:

• email module (gmail and others, just reading, sending can open up  a new window in the email client.)
• twitter module (multiple accounts…send and receive tweets)
• a twitter favorites module that allows my to follow the favorited tweets of various twitter accounts
• instapaper module for items to be read
• remember the milk  module for to-do items
• major site statistics module (only for my most active web properties)
• new incoming backlinks module
• paypal module for showing paypal balance and recent transactions
• google calendar module
• log monitor module (would tail the server logs for most active web properties)

What else would you add to the list? What data would you love to have at a click on your home page?

So, today I learned that Amazon S3 limits the amount of buckets you are allowed to have at any one time. This was a HUGE pain in the ass for me, as I just rewrote my S3 caching libraries to separate my caching into daily buckets.

The idea I had was that I would keep 30 days worth of buckets before automatically shuffling all those buckets into long term archival at the end of each month.

You see, Amazon S3 doesn’t allow for folders within buckets (not technically, I’ll get to that later), and I think it’s bloody ridiculous for me to store the ~15 millions xml cache files I need daily access to in just one massive bucket. So, I thought my plan to create daily buckets was pretty damn good. Apparently Amazon disagrees, as hey have set a limit on the number of buckets I can have associated with my account (FYI, I have ~90 buckets right now and that’s where the limit is).

So, I’ve been looking into my options, and I have decided that I’m going to have to go back to my original set up of storing all 15 million (and growing) xml files in one bucket. As some of you may know, S3 GUIs like S3Fox are able to show sub-folders within buckets, so I figure I will go this route. My first step was to take a crack at the Ruby S3 library from Amazon (not the AWS gem, which is crap for multi threaded environments) to see how I can create folders within buckets. Turns out, you can’t. At least, not out of the box. You see, AmazonS3 doesn’t actually SUPPORT sub-folders within buckets.

Stupid, right? I know.

So, how does S3Fox do it? It turns out they create virtualized folders by creating a special object that acts as a folder, then you access your stored objects by appending that folder object name to the actual file key. For a directory named “/foo”, you would create an object with the key “foo_$folder$”. Then, to get a directory listing of all files stored under the foo path, you just query S3 for objects with keys that start with “/foo”, and you ignore any objects that end with “_$folder$”.

I’m about to waste my day setting this up, and I’m none to happy about it. It seems like a hackish and shitty work around for an obvious service flaw. I’m sure there is some technically-reasonable answer for why Amazon has set an arbitrary limit for buckets and also why they don’t allow you to create folders, but I don’t know what it is. Anyone have any answers?

Hat-Tip the Dead Programmer Society for the code to create virtualized folders in S3.

The code was poorly presented, and poorly formatted. It wasn’t at ALL my normal caliber of work, so I went ahead and rewrote the concept into a tighter little wrapper.

I am using the Gattica gem to interact with the Google Analytics API.

This little script provides a wrapper around the gem that easily allows you to extract your sites organic keywords and sort them based on the number of unique entrances to your site via each keyword.

I thought I’d show you all a quick crash introduction on how you can access the API using Ruby.

The following code uses Open-Uri to access the API and Hpricot to parse the response. In this example, we’re going to authenticate with the API, and then we’re going to use it to extract all the keywords (organic and paid) which have brought visitors to our site within the past year (limited to 100 keywords max)

NB: sorry for the shitty formatting. I tried like hell to get this code to display well, but I gave up. Just copy and paste..you’ll be able to sort it out.

require ‘rubygems’
require ‘open-uri’
require ‘hpricot’

USER_EMAIL=”XXXXXX” #Insert your Google Account email address here
USER_PASS=”XXXXXX” #Insert your password here
PROFILE_ID=”XXXXXX” #Insert your profile ID here (Find this in your analytics profile under profile settings)
results = {}
keywords = []
auth_url = “https://www.google.com/accounts/ClientLogin?Email=#{USER_EMAIL}&amp;Passwd=#{USER_PASS}&amp;accountType=GOOGLE&amp;source=open-uri-limbo-v01&amp;service=analytics”

auth_key = (open(auth_url).read).split(”\n”).last

#now query for keywords, you can change the date values as you see fit, as well as the max-results value
response = Hpricot.XML(open(”https://www.google.com/analytics/feeds/data?start-date=2008-10-01&amp;end-date=2009-04-21&amp;dimensions=ga:keyword&amp;max-results=100&amp;ids=ga:#{PROFILE_ID}&amp;prettyprint=true”,
“Authorization” =&gt; “GoogleLogin #{auth_key}”).read)
## storing some of the extra information in a “results” hash in case we need it later
results[:total_results] = (response/’openSearch:totalResults’).inner_text
results[:start_index] = (response/’openSearch:startIndex’).inner_text
results[:items_per_page] = (response/’openSearch:itemsPerPage’).inner_text
results[:start_date] = (response/’dxp:startDate’).inner_text
results[:end_date] = (response/’dxp:endDate’).inner_text
results[:aggregates] = (response/’dxp:aggregates’).inner_text
results[:data_source] = (response/’dxp:dataSource’)
results[:entries] = (response/’entry’)
## the entries contain each individual keyword

## for my purposes, I’m not interested in visitors coming from search operators like “site:”
## so we’re scrubbing those out.
for entry in results[:entries]
keyword = (entry/’dxp:dimension’).first[:value]
keywords &lt;&lt; keyword unless keyword.include? “site:” or keyword.match(/\(.*\)/)
end

## Now print out our keywords
keywords.each {|kw|
puts kw
}

UPDATE just seconds ago, a new gem was released called Gattica which you can use to access the Google Analytics API. How’s that for up to the minute blogging? Thanks Twitter!

Web Applications are EASY to build! Anyone can do it!

First, you need an idea. This can be the hardest part. Basically, you just want to think about something people do every day that you can do better.
Here are some ideas to get you started:

• web-based makeup mirror (requires a webcam)
• sea-monkeys…give them facebook pages, make them twitter…put them up for adoption.
• a calculator, with lots and lots of ajax….and really big buttons.

These are just a few ideas, but you get the picture. Take some time coming up with your idea, and be patient! You have to refine the idea before you waste time building something no one will ever use. Take at least 30 minutes to decide.

Once you’ve decided on what your highly profitable Web App is going to do, you have to build it! This is the easiest part. You don’t even have to know how to write code. Really! There are new programming languages that don’t even require complicated things like squiggly brakets {} or dollar signs $.  Thanks to the magic of Ruby on Rails, if you can write a grocery list, you can write a Web App. Seriously!

Do you know how to view the source code for an HTML page? Great! That means you know how to design a website! Now, I know, designing things is complicated, but don’t worry about it! NO ONE DESIGNS THEIR OWN WEBSITES ANYMORE! Seriously! All you need is a Template. There are even WEB APPS to build Templates for WEB APPS! Totally meta! Gnarly!

So, you’ve got your ruby, you’ve got your template….you’re almost done! Now, you need a LOGO! Most people say you should hire a designer to get a Logo done professionally, but you don’t want to spend any money out of pocket, do you? I mean, what are you, Mother Teresa?!? Hell no! I recommend you just rip one off for now, and once you make some revenue, you can buy some cheap stock Logo from istockphoto.com.

Remember, in order to maximize Web App profits, you need to always be vigilant so that no corners escape uncut. When building your Web App, you should be sure to try and burden yourself with as many of the details as possible. You need to be prepared to be the programmer, the tester, the designer, the marketer and the support staff, ALL IN ONE! But don’t get stressed out about handling a big workload, because it’s not like you have to do ANY of those things at 100% efficiency, as long as you’re the only one doing them. The people who will use your Web App are just nameless, faceless drones anyways; it’s not like you owe them timely responses to their emails. Always remember that your users are eternally indentured to you in exchange for access to your Web App.

OK, so you’ve got your Template, you’ve got your Logo……..it’s time to do some programming! First, you want to download as many plugins and gems as possible. The more the better, because the more gems you have, the less code you have to write. Remember when your teacher crushed your dreams and told you that there are no unique ideas left, that everything has already been done? Well, when it comes to Gems, its true! There is a Gem or Plugin for every possible thing you could ever conceive of wanting to do – now, or in the future. So basically, just get as many Gems and Plugins as possible, and just start plugging them in wherever you need content.

Look at you! You go-getter! You’ve just programmed your first Web App! It’s going to be the BEST CALCULATOR EVER! Don’t worry about scaling it up. If your users ever start calculating pi, you can just use the Cloud, man! The Cloud is where it’s at! In case you haven’t heard of the Cloud yet, I’ll tell you a bit about it. The Cloud is this place in the Amazon Basin, a humid, godless, no-mans-land inhabited by under paid savages (Amazonians). Anyways, the Cloud is like this giant parking garage, but instead of cars, they’ve parked hundreds and hundreds of servers which they’ve woven together with millions of long, tangled fibreoptic wires (hence the name ‘the Cloud’). Anyways, you just email your program to the Amazonians, and they’ll put your program on a piece of the cloud, giving you the ability to add as many mongrel instances to your server cluster as your heart desires. Scaling?!?! What Scaling?!?!

Now that you’ve got your Template, and your Logo and your Program on the Cloud, you’re ready to start marketing!!! This part is really really easy. You just need to stir the pot a little bit, & kick up a little media interest. For example, let’s say one of your Virtual Sea-Monkeys ends up having to see a Substance Abuse Counselor…..that’s marketing gold! Big-mouthed bloggers love stories like that! The more sensational the better, especially because people will put your media story on sites like Digg, or StumbleUpon. These sites will send you visitors who will really take the time to read your story, visit your site, interact with your product, and develop long-lasting, meaningful connections with your brand. You just have to keep your eyes open for marketing opportunities!

By now, thanks to your Web App, you’re probably well on your way to dealing with your delinquent mortgage payments and are no longer dodging your relatives. Congratulations! Feel free to send your former boss a cruel and insensitive e-card from someecards.com, because You’ve Arrived!!! You’re a big web success story! Expect to hear from Inc. Magazine within minutes.

But! But! But! Don’t stop there! This is just the beginning! Now that you’ve seen how easy it is to create one Web App, create more! Don’t worry about the Web Apps that you’re already managing, it’s OK to let them slide. If you want, you can start an affiliate program and get affiliates to keep marketing your old, stale Web Apps while you’re busy building new ones! You’ll want to get an Email Autoresonder so that you’re not burdened by time consuming support requests to have subscriptions canceled, you’ve got better things to do! You’re an entrepreneur!

Google Web Search Ajax

I’ve heard more than a few people asking if this will stop automation software from crawling Googles SERPs to retrieve rankings or Adwords data. In short, the answer is no.

I doubt this development would be solely intended to stop people from using automated rank checkers or content scrapers, but Google has to know that whatever they change, people will simply adapt and evolve their software to keep pace. My personal opinion is that a move to ajax SERPs would provide google with a tighter control of how they serve data, as well as providing them with a whole slew of new metrics they could leverage. On the flip-side, this move could hamper the accessibility of Googles service from outdated/underpowered computers and browsers.

Regardless, the possibility of Google going ajax does indeed raise some questions in so far as how people should proceed with developing automation software. Not only is ajax delivered content more difficult to automate, but ajax also could be used to track mouse movement. This would be one of the metrics that ajax would make available to Google, and also it could be a possible means by which Google could begin to distinguish bots from humans. (although, on the Google-scale, that would be an incredibly large amount of data to process.) While AJAX content is not as trivial to scrape as traditional content, it is still quite possible. I think that as Google evolves to develop ways to clean up their SERPs, programmers and marketers will evolve as well. I believe that software will inevitably evolve to mimic human browsing behavior. This includes filling out form data, mouse movement, sending Google Toolbar data, using back buttons, clicking links, storing cookes, etc. And don’t forget, all that will have to be done at human speeds, not computer speeds. The requirement to mimic human behavior also means that it will become more and more difficult to multi-thread and do simultaneous requests from one IP range. Rather than managing massive proxy farms, in the future, it will be more cost-effective and productive to off-load automation requests to client computers which, when required, send data back to a master server for processing (no, I’m not talking about malware here….more like browser plugins).

There are already a slew of different libraries which mimic browsing behavior and also handle ajax. Off the cuff, ScrubyT handles ajax quite well. Watir can actually open an instance of IE, FF, Safari, or even Chrome, and thus fully mimic a browser. Marketers and programmers can use these libraries, and other libraries to both mimic human browsing behavior and make ajax calls to automate SERPs. In the future, you can expect these kinds of libraries to evolve and become more advanced and customized towards mimicking human behavior. Even still, if ajax calls are not encoded, you can simply extract the appropriate call out of the page and make it yourself. An example of this can be seen here.

At the end of the day, a move on Google’s part to ajax SERPs may throw a wrench into some peoples software, but it will certainly not herald an end to automation tools like Aaron Wall’s excellent SEOToolbar, or others.

I was recently doing a little friendly consulting for a friend of mine who runs a membership based adult-site.

Basically he was converting well but not getting alot of traffic, so we took some time to look over his sites layout and went over some basic SEO best-practices.

One of the things that he really needed to improve on was the layouts of his images. I mean, with a site like this, images are way more prevalent than content…his design is hugely image intensive, and there are literally hundreds of individual preview galleries that could potentially act as great landing pages for each individual sub-niche…

I was advising him that he should be working on keyword rich, unique captions and alt-tags for his images, and his complaint was that there are so many images that it would be a huge pain in the ass to go through and create captions/ALTs for each one. So, we decided that the best thing to do in this case would be to create a simple function that dynamically builds unique captions and ALT text for him.

Basically, what we did was build 10 different sets of four-to-five arrays containing various keywords that we could randomly choose from in building a complete string. We then set the function to dynamically choose one of the array sets, so that each caption/ALT didn’t have the exact same format.

In building each array set, we started with a sentence structure he wanted to use, and then expanded each word out into it’s own array.

Example: (I normally keep this blog PG13, so please excuse the dirty words, but I’m using an example relevant to the adult industry)

ARRAY1. Sentence structure:
“Download <model_name>’s private sex video”

array1a = [download,check out,view,see,get,look at]
array2b = [private,personal,intimate]
array3c = [sex,solo,masturbation,jerk-off,porn]
array4d = [video,film,clip,download,stream,flick,vid,picture]

ARRAY2. Sentence structure:
“<model_name> has tons of hot videos”

array2a = [has,uploads,provides,creates,films,makes]
array2b = [tons of, lots of, many of, hundreds of, a lot of, loads of]
array2c = [hot, sexy, steamy, erotic, raunchy, amazing, mind-blowing]
array4d = [videos,films,clips,downloads,streams,flicks,vids,pictures]

ARRAY3…etc..ARRAY4….etc…..all the way to TEN different sets of arrays.

The above is just an example….in a real production usage, I would recommend no less than 10-15 words PER array.

Then, for each caption and each alt, the webpage calls the function which randomly chooses one of the 10 array sets and uses it to build a dynamic string.

Once you’ve randomly selected one of the 10 array sets to use, you’d build the dynamic string like this:

dynamic_string = array1a[rand(array1a.length)]." ".model_name."'s ".array1b[rand(array1b.length)]." ".array1c[rand(array1c.length)]." ".array1d[rand(array1d.length)];

So, this function might return the following: “download Jenny’s intimate sex video” OR “checkout Suzie’s private porn clip” OR etc, etc etc

Using this simple and well-documented technique, the site owner was able to create keyword rich, dynamic captions/ALTs for every single picture in his hundreds of preview galleries. He was able to keep his image-rich site design and still put lots of great content into it without making it look like keyword spam, and all without going through the back-breaking effort of manually writing content.

Two closing notes fore you all to think about:

1)You can use your favorite keyword/synonym tools and even search query logs to build massive lists of your relevant keywords. Then you just need to take some time to scan through them and group and sort them into relevant sets.

2)This technique is very useful for anchor text as well

Situation:
Shared Host
CPanel
No SSH Access

Solution 1: FTP Transfer Folder  to server, Mechanize/CURL Install
Process:

1) Collect variables: database, username, cpanel login, cpanel password, keywords

2) If necessary, create databases and users on CPanel host (using CURL, or Mechanize)

Here is a function I use to create CPanel DBs:
def self.create_db(base_url, cpanel_loc, cpanel_user, cpanel_pass, db, dbuser, dbpass)
agent = WWW::Mechanize.new
agent.user_agent_alias = $USER_AGENTS[rand($USER_AGENTS.length)]
agent.basic_auth(cpanel_user,cpanel_pass)
url = “http://#{base_url}#{cpanel_loc}/frontend/x/sql/”
agent.get(url)
agent.get(”#{url}adddb.html?db=#{db}”)
agent.get(”#{url}adduser.html?user=#{dbuser}&pass=#{dbpass}”)
agent.get(”#{url}addusertodb.html?user=#{cpanel_user}_#{dbuser}&db=#{cpanel_user}_#{db}&ALL=ALL”)
end

3) For each keyword in the keyword list, create a different blog

3a) Write local wordpress-config file with user entered data

3b) Pull a random theme from local theme library

3c) Pull required plugins from local plugin library

3d) Create local folder for blog, copy base wordpress install files, plus files from 3a,3b,3c

4) Begin FTP Transfer of files from master server to destination on install server (Thread this process to speed up)

5) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate theme selected in 3b and plugins selected in 3c

*Repeat step 3 for each keyword supplied to the function

NOTES: This method, when optimized and threaded, can install about one blog / 90 seconds. The major speed limiting factor is the FTP transferring. A base wordpress install already has a TON of files, and depending on how many plugins and how many themes you are including, the number of files really slows down the FTP transfer. I recommend creating a personal WP base install with ALL of the junk and cruft removed (ie: the management panel niceities like password strength checking etc..). Removing all non-essential files from a wordpress base install can really speed up FTP transfers. This is a good practice anyways, since we want to be as resource efficient as possible, since we’re going to cram this shared server with nothing but blogs.

Solution 2: FTP package file to server, extract through file manager, Mechanize/CURL Install
Process:

1) Collect variables: database, username, cpanel login, cpanel password, keywords

2) If necessary, create databases and users on CPanel host (using CURL, or Mechanize)

3) For each keyword in the keyword list, create a different blog

3a) Write local wordpress-config file with user entered data

3b) Pull a random theme from local theme library

3c) Pull required plugins from local plugin library

3d) Create local folder for blog, copy base wordpress install files, plus files from 3a,3b,3c

3e) Package folder

4) FTP Transfer Package from master server to destination on install server

5) Mechanize or CURL CPanel Filemanager to unzip file on server

6) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate theme selected in 3b and plugins selected in 3c

*Repeat step 3 for each keyword supplied to the function

NOTES: This method, when optimized and threaded, can install one in about 25 seconds. Once again, the major impediment to speed here is the FTP transfer. This is still a lot faster than Solution 1, and is perhaps the best solution for a situation where you can’t use SSH. I should note that you could also just create one central tarball and then mechanize/CURL the CPanel File Manager to copy and unzip it for each new install, but…I really hate to use that amount of screen automation. It just tends to break down so often.

Situation:
Shared / VPN Server
CPanel
SSH Access

Solution 1: Master Server Central Wordpress Install / Mechanize CURL Install
Process:
1) Upload base Wordpress install files, Theme Library, Plugin Library from master server to central location on install server
2) Create a deployment file that does the following:
2a) Copy base Wordpress install to destination folder using SSH
2b) Set up symbolic links between destination install themes & plugins, with the central libraries uploaded in step 1.
2c) If necessary, create databases and users
2d) Create Wordpress Config file based on information supplied by master server
3) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate desired theme & plugins.
* For each blog you wish to install, just upload a new deployment file

NOTES: Once you have installed the central libraries, this method is extremely fast.

CONCLUSIONS: At the end of the day, it is definitely faster and more space efficient if you have SSH access through which you can automate the install process. Unfortunately, many shared hosts don’t offer SSH access (some offer it only if you provide some form of government identification). Many people go for shared hosting situations because they are very cheap, low-risk and you can afford to buy many different hosting packages with different IP addresses. For people who are running more sophisticated networks and have greater resources, VPS situations (without CPanel) are preferable, since they are more stable with more available resources, and of course, SSH, which really makes automation a breeze.

Automating the deployment and install of Wordpress blogs can be a really frustrating process to wade through. That said, ..blah….blah…and…gratifying….blah…1000 blogs….blah blah…button click….bla
h…elite spammer…blah blah