So, today I learned that Amazon S3 limits the amount of buckets you are allowed to have at any one time. This was a HUGE pain in the ass for me, as I just rewrote my S3 caching libraries to separate my caching into daily buckets.

The idea I had was that I would keep 30 days worth of buckets before automatically shuffling all those buckets into long term archival at the end of each month.

You see, Amazon S3 doesn’t allow for folders within buckets (not technically, I’ll get to that later), and I think it’s bloody ridiculous for me to store the ~15 millions xml cache files I need daily access to in just one massive bucket. So, I thought my plan to create daily buckets was pretty damn good. Apparently Amazon disagrees, as hey have set a limit on the number of buckets I can have associated with my account (FYI, I have ~90 buckets right now and that’s where the limit is).

So, I’ve been looking into my options, and I have decided that I’m going to have to go back to my original set up of storing all 15 million (and growing) xml files in one bucket. As some of you may know, S3 GUIs like S3Fox are able to show sub-folders within buckets, so I figure I will go this route. My first step was to take a crack at the Ruby S3 library from Amazon (not the AWS gem, which is crap for multi threaded environments) to see how I can create folders within buckets. Turns out, you can’t. At least, not out of the box. You see, AmazonS3 doesn’t actually SUPPORT sub-folders within buckets.

Stupid, right? I know.

So, how does S3Fox do it? It turns out they create virtualized folders by creating a special object that acts as a folder, then you access your stored objects by appending that folder object name to the actual file key. For a directory named “/foo”, you would create an object with the key “foo_$folder$”. Then, to get a directory listing of all files stored under the foo path, you just query S3 for objects with keys that start with “/foo”, and you ignore any objects that end with “_$folder$”.

I’m about to waste my day setting this up, and I’m none to happy about it. It seems like a hackish and shitty work around for an obvious service flaw. I’m sure there is some technically-reasonable answer for why Amazon has set an arbitrary limit for buckets and also why they don’t allow you to create folders, but I don’t know what it is. Anyone have any answers?

Hat-Tip the Dead Programmer Society for the code to create virtualized folders in S3.

I wanted to share some interesting tid-bits with you all from a recent Ipsos Reid presentation (June 18th) on Social Network Marketing.

This presentation was made to BC Hydro, the major Power authority in British Columbia (that’s in Canada, dummy).

The average Canadian:

• 5.4 hrs/week spent on Social network sites, about 1/3 of total Online time.
• LinkedIn captures 70% of the share of Social Networking in the workplace.
• The stuff teens do online is very different from adults, and VERY limited: socializing, music, and gaming.
  • 88% of Adults use the internet to visit a new or informational website, versus only 44% of teens.
  • 68% of Adults have clicked a website advertisement versus 28% of teens.
  • 70% teens are weekly social network users, versus 36% of adults

Highlights:

• Advertising budgets are not aligned to where people are spending their time – online expenditure lags significantly behind traditional mediums
• 58% of Canadians receive 51+ unsolicited commercial emails a week (17% don’t even know)
  • 64% of people don’t open ANY unsolicited emails
  • of the people that do open unsolicited emails, 53% do so out of curiosity, 28% because they thought it was legitimate
• Permission-based (opt-in) emails are still one of the best ways to market to your websites audience. Be diligent about collecting them
• 77% of Canadians are registered to receive some kind of opt-in email (2008 average number of sites registered with: 15.3)
  • Top 3 reasons for opting-in: Personal Interest (42% ), Entertainment (38%), New and Information (32%)
  • It’s worth noting that E-Commerce and Retail is the 7th most common reason (27%)
• Activities resulting from opting-in to an email list:
  • 60% entered an advertiser’s contest
  • 52% visited the advertiser’s website
  • 17% purchase or received products at a later date
• 68% percent of Canadians are willing to provide their email address depending on reasoning

I’ve uploaded the full presentation for you all, which can be viewed here:

Google Web Search Ajax

I’ve heard more than a few people asking if this will stop automation software from crawling Googles SERPs to retrieve rankings or Adwords data. In short, the answer is no.

I doubt this development would be solely intended to stop people from using automated rank checkers or content scrapers, but Google has to know that whatever they change, people will simply adapt and evolve their software to keep pace. My personal opinion is that a move to ajax SERPs would provide google with a tighter control of how they serve data, as well as providing them with a whole slew of new metrics they could leverage. On the flip-side, this move could hamper the accessibility of Googles service from outdated/underpowered computers and browsers.

Regardless, the possibility of Google going ajax does indeed raise some questions in so far as how people should proceed with developing automation software. Not only is ajax delivered content more difficult to automate, but ajax also could be used to track mouse movement. This would be one of the metrics that ajax would make available to Google, and also it could be a possible means by which Google could begin to distinguish bots from humans. (although, on the Google-scale, that would be an incredibly large amount of data to process.) While AJAX content is not as trivial to scrape as traditional content, it is still quite possible. I think that as Google evolves to develop ways to clean up their SERPs, programmers and marketers will evolve as well. I believe that software will inevitably evolve to mimic human browsing behavior. This includes filling out form data, mouse movement, sending Google Toolbar data, using back buttons, clicking links, storing cookes, etc. And don’t forget, all that will have to be done at human speeds, not computer speeds. The requirement to mimic human behavior also means that it will become more and more difficult to multi-thread and do simultaneous requests from one IP range. Rather than managing massive proxy farms, in the future, it will be more cost-effective and productive to off-load automation requests to client computers which, when required, send data back to a master server for processing (no, I’m not talking about malware here….more like browser plugins).

There are already a slew of different libraries which mimic browsing behavior and also handle ajax. Off the cuff, ScrubyT handles ajax quite well. Watir can actually open an instance of IE, FF, Safari, or even Chrome, and thus fully mimic a browser. Marketers and programmers can use these libraries, and other libraries to both mimic human browsing behavior and make ajax calls to automate SERPs. In the future, you can expect these kinds of libraries to evolve and become more advanced and customized towards mimicking human behavior. Even still, if ajax calls are not encoded, you can simply extract the appropriate call out of the page and make it yourself. An example of this can be seen here.

At the end of the day, a move on Google’s part to ajax SERPs may throw a wrench into some peoples software, but it will certainly not herald an end to automation tools like Aaron Wall’s excellent SEOToolbar, or others.

Currently we have two big ultra-modern leather couches in our media room. We have been thinking that they’re a bit too stiff, a bit too cold. Basically, you have to be a contortionist to perform any kind of cuddling while watching a movie…so…it was time for a new couch.

In the first store we stopped at, we immediately saw a modular sofa set that was perfect…it had modern leather-clad frame but big comfy fabric cushions, and throw pillows instead of stiff leather cushions for the back…it was, in short, the ideal couch for us; especially since we were able to buy separate pieces and put them together in the arrangement we wanted (I love modular furniture). So, long story short, after we checked the dimensions, we were a firm sale (probably the easiest sale our salesman got all day).

Now, when we were checking out and paying for the sofa, the salesman did something that got me thinking: He didn’t up-sell us. He didn’t even TRY to up-sell us into add-ons, warranties, teflon stain-proofing, etc etc. No, he didn’t up-sell us, because he just put it all on the bill anyways without even asking us. Basically, he breezed over the itemized list on the bill and drew our attention directly to the total and said, “so that will be $____ dollars.” Now, I had been calculating the whole thing in my head, so I knew what to expect in terms of a final total. When I heard his total, I said, “Let me see that bill….O.K, you’ve put Teflon on here for 25.99 per module, coming to 181.00.” and he says, “Oh you want that, you definitely want that. I put it on there because you’d be crazy not to want that.” And Randy, standing next to me, said “Oh yeah, let’s get that! You’re always spilling your wine!” (sadly, it’s true). So I went on down the bill to the next item, “Now, whats this 125.00 steel base item?” and the salesman says “Well, of course, you want the steel base right? It looks a lot better than the wood. Everyone is going steel nowadays.” And Randy was there saying “Oh yeah! I love that style.” So, we got the steel base, too….

This debacle went on, item by item, until I was completely defeated and had no choice but to accept every add-on item on that bill except for one thing: the rug underlay (no joke, that was on the bill…) After we paid up and left – my wallet hurting alot more than I expected it too – I said, “We certainly bought alot more than we bargained for”, and Randy replied, “Well we knew it was going to be expensive, and besides, we didn’t end up paying as much as he initially tried to bill us.”

And there it was, the brilliance of not up-selling your customers. We never even saw what the price would have been without all those add-ons, we only saw the total price with all the add-ons included. The psychology of the situation is that once a customer sees an expensive base price, their wallet tightens up and they don’t want to spend anything more, even if it is just 25.99 per module for teflon coating, or 125.00 for a steel base.

Think about what would be more difficult: showing your customer a total price and then convincing them to spend more on upgrades? or showing a customer an upgraded price and letting them remove what they don’t really want? The answer is obvious. Chances are, the latter method will tend to get more customers to spend money on upgrades than the former method.

This lesson applies to the web as well, especially if you are marketing any kind of web-application. If you are selling something that has upgrades, why not include the upgrades automatically at checkout? It’s a win-win situation: The customer always feels better when they see their total price decrease as they customize their purchase to their needs, and chances are you’ll sell more upgrades than usual.

The techniques and discipline applied to researching memes can also be applied to brands and niche specific search queries. Its useful to research this kind of information because it lets you know where your audience is communicating. Knowing where your brand/niche evangelists are communicating is an incredible asset, because they really have their fingers on the pulse of that brand/niche, and people LISTEN to them. By knowing where the evangelists communicate, you are in a position to deal with bad PR before it spins out of control; you’re also in position to exploit and leverage opportunities you might have missed if you were just relying on Google News Alerts, and BlogSearch.

This isn’t just about researching the buzz on social sites like Twitter. I mean, let’s face it, Twitter has a fairly narrow demographic. For example, let’s say you want to know how to best communicate to real estate investors. You better know where the realtors are hanging out on the web. You’ve got to get your hands dirty and identify the popular opinion blogs, the obscure dicussion forums, and so on.

I use alot of the obvious tools to do my research, like google search, blog search, technorati, etc, but, my favorite new tool is BoardReader. It is, hands down, one of the best research tools I’ve used in recent months.

Let’s say I wanted to find out where people are talking about condos in Toronto. I’d goto BoardReader and peform this search

With that one search, I’ve just been given a whole group of new websites to research and monitor. Now I know that people are talking about Toronto Condos on forums such as SkyScraper Forum, and InvestorsHub  and Realestatetalks.com and Cyburbiaforums etc etc etc.

And the best part of this is that BoardReader gives us a list of recent/relevant topics and posts on these forums. This makes it incredibly easy for us to monitor the conversations, find the opinion makers, and connect with our audience through meaningful, relevant and timely communications.

I got the idea when I found some domains that ranked for various viagra search terms back in 2001 that are now unregistered. They are old, plus have backlinks, which is a double bonus. We all know how much Google loves old domains!

Anyways, I created a small script that goes through the 2001 SERPs and pulls each domain into a list that is run through a whois search to see if it’s available. In under and hour I found almost 15 great domains. The thing that I like about this, compared to domain auctions is that you know you’re buying a domain that is relevant to your niche! And chances are, whatever backlinks those domains still have will also be relevant to your niche.

Give it a whirl! Some of you might be pleasantly surprised with what you find.

Today, there is a great example of how valuable TwitScoop is. It seems that Kanye West got arrested for beating on some paparazzi…have you heard about that yet? If you are relying on Google Trends, then no, you haven’t, because it hasn’t hit Google Trends yet. But its already all over the front page of TwitScoop. You can bet that Kanye West Arrested will be all over the front page of Google Trends in about 12 hours. This is because Twitter has its pulse on exactly what people are talking about at this exact moment, whereas Google Trends represents what people were searching for 12-24 hours ago.

So, next time you’re researching what the moment’s hot topics are, check out TwitScoop first, and Google Trends second.

OH, and PS: TwitScoop has an API….so…yeah…..it’s a slam dunk.

Situation:
Shared Host
CPanel
No SSH Access

Solution 1: FTP Transfer Folder  to server, Mechanize/CURL Install
Process:

1) Collect variables: database, username, cpanel login, cpanel password, keywords

2) If necessary, create databases and users on CPanel host (using CURL, or Mechanize)

Here is a function I use to create CPanel DBs:
def self.create_db(base_url, cpanel_loc, cpanel_user, cpanel_pass, db, dbuser, dbpass)
agent = WWW::Mechanize.new
agent.user_agent_alias = $USER_AGENTS[rand($USER_AGENTS.length)]
agent.basic_auth(cpanel_user,cpanel_pass)
url = “http://#{base_url}#{cpanel_loc}/frontend/x/sql/”
agent.get(url)
agent.get(”#{url}adddb.html?db=#{db}”)
agent.get(”#{url}adduser.html?user=#{dbuser}&pass=#{dbpass}”)
agent.get(”#{url}addusertodb.html?user=#{cpanel_user}_#{dbuser}&db=#{cpanel_user}_#{db}&ALL=ALL”)
end

3) For each keyword in the keyword list, create a different blog

3a) Write local wordpress-config file with user entered data

3b) Pull a random theme from local theme library

3c) Pull required plugins from local plugin library

3d) Create local folder for blog, copy base wordpress install files, plus files from 3a,3b,3c

4) Begin FTP Transfer of files from master server to destination on install server (Thread this process to speed up)

5) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate theme selected in 3b and plugins selected in 3c

*Repeat step 3 for each keyword supplied to the function

NOTES: This method, when optimized and threaded, can install about one blog / 90 seconds. The major speed limiting factor is the FTP transferring. A base wordpress install already has a TON of files, and depending on how many plugins and how many themes you are including, the number of files really slows down the FTP transfer. I recommend creating a personal WP base install with ALL of the junk and cruft removed (ie: the management panel niceities like password strength checking etc..). Removing all non-essential files from a wordpress base install can really speed up FTP transfers. This is a good practice anyways, since we want to be as resource efficient as possible, since we’re going to cram this shared server with nothing but blogs.

Solution 2: FTP package file to server, extract through file manager, Mechanize/CURL Install
Process:

1) Collect variables: database, username, cpanel login, cpanel password, keywords

2) If necessary, create databases and users on CPanel host (using CURL, or Mechanize)

3) For each keyword in the keyword list, create a different blog

3a) Write local wordpress-config file with user entered data

3b) Pull a random theme from local theme library

3c) Pull required plugins from local plugin library

3d) Create local folder for blog, copy base wordpress install files, plus files from 3a,3b,3c

3e) Package folder

4) FTP Transfer Package from master server to destination on install server

5) Mechanize or CURL CPanel Filemanager to unzip file on server

6) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate theme selected in 3b and plugins selected in 3c

*Repeat step 3 for each keyword supplied to the function

NOTES: This method, when optimized and threaded, can install one in about 25 seconds. Once again, the major impediment to speed here is the FTP transfer. This is still a lot faster than Solution 1, and is perhaps the best solution for a situation where you can’t use SSH. I should note that you could also just create one central tarball and then mechanize/CURL the CPanel File Manager to copy and unzip it for each new install, but…I really hate to use that amount of screen automation. It just tends to break down so often.

Situation:
Shared / VPN Server
CPanel
SSH Access

Solution 1: Master Server Central Wordpress Install / Mechanize CURL Install
Process:
1) Upload base Wordpress install files, Theme Library, Plugin Library from master server to central location on install server
2) Create a deployment file that does the following:
2a) Copy base Wordpress install to destination folder using SSH
2b) Set up symbolic links between destination install themes & plugins, with the central libraries uploaded in step 1.
2c) If necessary, create databases and users
2d) Create Wordpress Config file based on information supplied by master server
3) Mechanize/CURL to appropriate Install URL, then Mechanize/CURL through management panel to activate desired theme & plugins.
* For each blog you wish to install, just upload a new deployment file

NOTES: Once you have installed the central libraries, this method is extremely fast.

CONCLUSIONS: At the end of the day, it is definitely faster and more space efficient if you have SSH access through which you can automate the install process. Unfortunately, many shared hosts don’t offer SSH access (some offer it only if you provide some form of government identification). Many people go for shared hosting situations because they are very cheap, low-risk and you can afford to buy many different hosting packages with different IP addresses. For people who are running more sophisticated networks and have greater resources, VPS situations (without CPanel) are preferable, since they are more stable with more available resources, and of course, SSH, which really makes automation a breeze.

Automating the deployment and install of Wordpress blogs can be a really frustrating process to wade through. That said, ..blah….blah…and…gratifying….blah…1000 blogs….blah blah…button click….bla
h…elite spammer…blah blah

I Could Be Anything monitors the Twitter public timeline and looks for messages that can be appended to the words “I am”.

Back when I was doing my degree, I used to create alot of work that involved typography and poetry, so this was a fun little site to build. I played around alot with how to filter out certain stop words (he, she, etc), and punctuation marks, also I discovered that typically if a sentence fragment has more than four words between “the” and a punctuation mark, it won’t make sense when appeneded to “I am”.

Anyways, this might all seem pretty mundane, but it makes for great research. Twitter provides unlimited data for me to work with, and whenever I have time, I’ll continue tweaking and trying to perfect the filters.

For example, lets say I had four word groups:

- pronouns (my, your, his, her, our)

- subjects (goat, sailboat, cabin)

- conjunctions (is, is usually, is always)

- adjectives (stinky, leaking, clean)

By analyzing all the possible permutations of these 4 word groups, we can see exactly how much content we can spin from it, and what all the possible permutations are.

In this case, there are 135 possible permutations:

my goat is stinky
my goat is leaking
my goat is clean
my goat is usually stinky
my goat is usually leaking
my goat is usually clean
my sailboat is usually leaking
my sailboat is usually clean
our cabin is always leaking
our cabin is always clean
etc…etc….etc…..

The possible applications of string permutation become apparent when you consider a real-world example, like creating headlines:

So You Want To Meet a Girl
So You Want To Get A Date
So You Want To Ask Out A Woman
So You Want To Ask A Woman Out
So You Want To Get Started Dating
Learn How To Meet a Girl
Learn How To Get A Date
Learn How To Ask Out A Woman
Learn How To Ask A Woman Out
Learn How To Get Started Dating
Find Out The Way To Meet a Girl
Find Out The Way To Get A Date
Find Out The Way To Ask Out A Woman
Find Out The Way To Ask A Woman Out
Find Out The Way To Get Started Dating
It’s Easy To Learn How To Meet a Girl
It’s Easy To Learn How To Get A Date
It’s Easy To Learn How To Ask Out A Woman
It’s Easy To Learn How To Ask A Woman Out
It’s Easy To Learn How To Get Started Dating

When I first started analyzing the possible permutations of some of my madlib scripts, I was so astounded by the numbers that I assumed I had made a mistake in my coding. After I did the math manually, I realized that I had it right, but the numbers were still astounding.

I have one relatively short madlib script that outputs around 150 words. However, once you take into account number of possible rows in each database it draws from, plus the numerous wildcard sets, that script has approx 42311189564227584000 possible unique permutations.

To give you an idea of how steep the corelation is between input and output, here are some numbers.

Each line represents one word group.
10 <– first group has 10 words
20 <– second group has 2 words
100 <– third group has 5 words
1300 <– fourth group has 13 words
20800 <– etc etc etc
166400
998400
10982400
54912000
219648000
1537536000
4612608000
27675648000
110702592000
332107776000
996323328000
3985293312000
15941173248000
95647039488000
382588157952000
2295528947712000
9182115790848000
36728463163392000
110185389490176000
330556168470528000
1322224673882112000
5288898695528448000
10577797391056896000
42311189564227584000

For all you programmers out there, here is how I am calculating permutations (this needs to be threaded to have any production application):

class Array
def sequence(i = 0, *a)
return [a] if i == size
self[i].map {|x|
sequence(i+1, *(a + [x]))
}.inject([]) {|m, x| m + x}
end
end

input = [
["my","your","her","his","our"],
["goat","sailboat","cabin"],
["is","is usually","is always"],
["stinky","leaking","clean"]
]

output = input.sequence

p output.length
p output