• The default Ubuntu 12.04 LTS AWS image boots on a 8GB  root partition created on EBS. This allows you to run a a normal Ubuntu server. If you create a larger EBS partition, Ubuntu will automatically expand the file system on first boot, but 8GB should be plenty for a typical DNS server.
• The Micro Instances includes 600MB of RAM
• I thought I read that the Micro Instance includes 2 CPU cores, but the image I booted only shows 1 CPU.

This should be fine for a backup DNS server.

The AWS documentation is large and intimidating. In the past, AWS encouraged a lot of specialized AWS development to create paravirtualized kernels that worked under Xen combined with VM instances that booted up and automatically configured  themselves to use S3 storage. This is a great model for web sites and other Internet based services that require scaling across many nodes, it seemed like overkill for something like DNS that typically only requires a couple of high availability servers. Fast forward several years later and now AWS lets you easily create VM backed by EBS storage that can be managed like a typical server. Here are the steps I followed to create my AWS based DNS server:

1. Opened an AWS account  - credit card is required (!)
2. Configure and start a new Ubuntu 12.04 LTS micro instance. The free instance is clearly identified and the defaults end up with an EBS backed server with a 8GB root partition.
3. Add a public IP address: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
4. Login to the new instance with the AWS generated SSH key pair: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html
5. Configure your new instance normally

Now we just have to wait and see if it is really free!

• http://en.wikipedia.org/wiki/Domain_tasting
• http://en.wikipedia.org/wiki/Domain_name_front_running

A friend forwarded me an email yesterday with a scam I haven’t seen before. He received domain name spam that looked like this:

Take time to secure friendsdomainnamewithans.com today.

We understand the importance of capturing all similar Domain Names in alignment with your business. Make an offer today in order for us to go after the domain name for you.
To confirm interest in owning this domain, fill out form here:

http://dr-gs.com/24343840mavidu-MCI

[...]

When I checked yesterday, the domain name was registered to a company with a name something like  “China Domains” (too bad I didn’t save the page). The domain was set to expire in a year. When I checked the domain name today, it was unregistered. It looks like somebody is combining spam with domain tasting… so creative!

• ElCap Report 6/17/08: Ian and I are the team of “American men”. We did make it to El Cap Tower
• ElCap Report 6/18/08: We didn’t catch the girls, but we did make it to Camp 5 by climbing late into the night.
• ElCap Report 6/19/08: We made it to the top.

Here are links to my photos of the climb:

• Pictures of our climb made with a disposable camera
• Pictures of our climb from the ground by El Cap photographer Tom Evans

After a few experiments, I discovered that the Panasonic was happy with .mkv files with H.264 encoding. After playing with several programs capable of transcoding on Ubuntu, I settled on Handbrake.

As many probably do, I had a large and messy collection of video files that needed to be converted. The one saving grace to the collection was that each group of files for a given movie was in a separate folder with a name that was based on the movie title plus other junk. Since I was cleaning things up anyway, I took some time to manually rename the folders in a consistent way. I did this in a way that allowed my transcoding program to parse the movie’s folder name and use it to name the resulting .mkv files. The result of all this is the simple Python program, transcode_avis.py that generates the Handbrake commands needed to convert .avi files with arbitrary video encoding to .mkv files encoded with H.264. The audio stream is copied rather than transcoded, as is my preference, to preserve original DVD digital surround sound streams when available.

I used transcode_avis.py to generate a batch file that I ran by hand. The batch file invokes HandbrakeCLI to do the work. I did it this way because I was going to slice up the the work into multiple batch files which I could run simultaneously to take advantage of my 8 core desktop CPU. But, this was unnecessary as Handbrake does a good job of using up all the cores on its own:

Transcoding 2 videos at once does push the CPU usage completely to 100% but it essentially transcodes each video at 1/2 speed.  The typical transcoding speed on my system is 265 frames per second (or 1/2 of that if transcoding 2 videos at once).

• Why Linq beats SQL
• IQToolkit (for building Linq data source of any type)
• Linq to IMAP (looks very useful and a very cool demonstration of the possibilities)

The G1 was almost good enough, but I heard it was slow and slow computers drive me crazy. Finally, my Droid has arrived. I’m not thrilled that Java is the recommended programming language, but at least you don’t have to program in Objective C. I’m on day 2 with my Droid and I’m really just getting started. The first challenge I ran into was moving my contacts from Thunderbird and my old Casio “Boulder” phone. With my old Casio phone, I used Verizon’s free Backup Assistant to keep my contacts backed up. This application is actually a nice touch by Verizon and they should publicize it better. After your first backup, you can login to their site and see your contacts. Last time I switched phones, I just installed Backup Assistant on my new phone and downloaded my contacts from my online backup. Unfortunately, Verizon has not yet ported Backup Assistant to the Android platform. They do, however, allow you to login to their web site and download all contacts as a CSV files. I though for sure there would be an easy way to get contacts in a CSV file onto my Droid, but that did not turn out to be the case.

My first task was to export another CSV from Thunderbird and consolidate my cell phone and email contacts. Luckily, this wasn’t too bad because I had done it last time I switched phones. I used OpenOffice’s Calc program to open both CSV file, manually consolidate them, then export the resulting consolidated contact list as a CSV file.

To get my consolidated CSV onto the Droid, first, I tried copying the CSV file onto the Droid and using the phone’s import contact function. With a USB cable,  getting the CSV file onto the phone was easy enough, but the Droid import function only understands vCards, so the CSV file was useless for the Droid import function.

The Droid will automatically sync with Gmail Contacts and I knew Gamail had a CSV import function, so I tried uploading my CSV file into Gmail Contacts. I could get some of the CSV uploaded, but, after many tries, I could not get Email addresses and mobile phone numbers properly imported. These are the most important data items, so IMO, Gmail’s CSV import is almost useless.

Next I tried to use an online service to convert my CSV to vCard, then import the resulting vCard file via the Droid contact import function. This sort of worked, but contacts with multiple phone numbers had all phone numbers set to be “home” and all email addresses were set as “other” (rather than “work” or “home”). I got the bright idea to import the vCard file into Gmail contacts, but this produced the same problems as trying to import directly on the Droid.

Thinking that getting contacts into Gmail must be a common problem, I ran a bunch more Google searches looking for something that would do the conversion I needed, but I found nothing.

So, I decided to do what only a programmer can do… write a program to get my contacts from the CSV onto my Droid. Actually, I did this indirectly. I used Google’s .Net data API to upload my CSV into Gmail Contacts, then the Droid’s built-in Gmail sync function automatically puts the contacts onto my Droid. I did it this way because a) I’ve worked the Google .Net data API before and found it easy to use and b) I have a good CSV library I’ve found and used on other projects.

For those interested, I have copied the full source code below:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

using Google.GData.Contacts;
using Google.GData.Client;
using Google.GData.Extensions;
using Google.Contacts;

using LumenWorks.Framework.IO.Csv;

namespace GoogleContactsUpload
{
    class Program
    {
        static void Main(string[] args)
        {
            string username = "user1@gmail.com";
            string password = "password1";

            Uri feedUri = new Uri(ContactsQuery.CreateContactsUri(username));

            RequestSettings rs = new RequestSettings("teztech-GoogleContactsUpload-1", username, password);
            ContactsRequest cr = new ContactsRequest(rs);

            using (StreamReader contactsReader = new StreamReader(@"C:\Users\pj\Documents\Databases\ContactsMerged.csv"))
            {
                using (CsvReader contactsCsv = new LumenWorks.Framework.IO.Csv.CsvReader(contactsReader, true, ',', '"', '"', '\0', true))
                {
                    while (contactsCsv.ReadNextRecord())
                    {
                        Contact contact = new Contact();
                        contact.Title = string.Format("{0} {1}", contactsCsv["First Name"], contactsCsv["Last Name"]);

                        if (contactsCsv["Email Address"] != "")
                        {
                            EMail workEmail = new EMail(contactsCsv["Email Address"]);
                            workEmail.Primary = true;
                            workEmail.Rel = ContactsRelationships.IsWork;
                            contact.Emails.Add(workEmail);
                        }

                        if (contactsCsv["Home Email"] != "")
                        {
                            EMail homeEMail = new EMail(contactsCsv["Home Email"]);
                            homeEMail.Rel = ContactsRelationships.IsHome;
                            contact.Emails.Add(homeEMail);
                        }

                        if (contactsCsv["Work Phone"] != "")
                        {
                            PhoneNumber workPhone = new PhoneNumber(contactsCsv["Work Phone"]);
                            workPhone.Rel = ContactsRelationships.IsWork;
                            contact.Phonenumbers.Add(workPhone);
                        }

                        if (contactsCsv["Home Phone"] != "")
                        {
                            PhoneNumber homePhone = new PhoneNumber(contactsCsv["Home Phone"]);
                            homePhone.Rel = ContactsRelationships.IsHome;
                            contact.Phonenumbers.Add(homePhone);
                        }

                        if (contactsCsv["Mobile Phone"] != "")
                        {
                            PhoneNumber mobilePhone = new PhoneNumber(contactsCsv["Mobile Phone"]);
                            mobilePhone.Primary = true;
                            mobilePhone.Rel = ContactsRelationships.IsMobile;
                            contact.Phonenumbers.Add(mobilePhone);
                        }

                        if (contactsCsv["Work Fax"] != "")
                        {
                            PhoneNumber faxPhone = new PhoneNumber(contactsCsv["Work Fax"]);
                            faxPhone.Rel = ContactsRelationships.IsFax;
                            contact.Phonenumbers.Add(faxPhone);
                        }

                        if (contactsCsv["Home Address"] != "")
                        {
                            PostalAddress homeAddress = new PostalAddress();
                            homeAddress.Value = FormatAddress(false, contactsCsv["Home Address"], contactsCsv["Home Address 2"], contactsCsv["Home City"], contactsCsv["Home State"], contactsCsv["Home ZipCode"], contactsCsv["Home Country"]);
                            homeAddress.Rel = ContactsRelationships.IsHome;
                            contact.PostalAddresses.Add(homeAddress);
                        }

                        if (contactsCsv["Work Address"] != "")
                        {
                            PostalAddress workAddress = new PostalAddress();
                            workAddress.Value = FormatAddress(false, contactsCsv["Work Address"], contactsCsv["Work Address 2"], contactsCsv["Work City"], contactsCsv["Work State"], contactsCsv["Work ZipCode"], contactsCsv["Work Country"]);
                            workAddress.Rel = ContactsRelationships.IsWork;
                            contact.PostalAddresses.Add(workAddress);
                        }

                        if (contactsCsv["Organization"] != "")
                        {
                            Organization organization = new Organization();
                            organization.Label = contactsCsv["Organization"];
                            organization.Title = contactsCsv["Job Title"];
                            contact.Organizations.Add(organization);
                        }

                        StringBuilder notes = new StringBuilder();

                        if (contactsCsv["Web Page 1"] != "")
                            notes.AppendFormat("Website: {0}", contactsCsv["Web Page 1"]);

                        notes.AppendFormat("{0}{1}", notes.Length > 0 ? "\r\n" : "", contactsCsv["Notes"]);

                        contact.Content = notes.ToString();

                        Contact createdContact = cr.Insert(feedUri, contact);
                    }
                }
            }
        }

        static public string FormatAddress(bool useBR, string address1, string address2, string city, string state, string zip, string country)
        {
            StringBuilder address = new StringBuilder();
            string newLine = useBR ? "<br />" : "\r\n";

            if(address1 != "") address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   address1);
            if(address2 != "") address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   address2);

            StringBuilder cityStateZip = new StringBuilder();

            cityStateZip.Append(city);
            if(state != "")    cityStateZip.AppendFormat("{0}{1}", cityStateZip.Length > 0 ? ", "  : "", state);
            if(zip != "")      cityStateZip.AppendFormat("{0}{1}", cityStateZip.Length > 0 ? "  " : "", zip);

            if(cityStateZip.Length > 0) address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   cityStateZip.ToString());
            if(country != "")           address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   country);

            return address.ToString();
        }

    }
}

It’s All About the Money

Many surveys and studies of programmers suggest that job satisfaction is more important than money, but my experience is that you have to take care of money issues first, or you’ll keep coming back to them over and over. The ideal situation is:

1. Everybody feels that they are being generously compensated (some of the generosity can be in deferred compensation - see #2)
2. Any type of deferred compensation (sales commissions, stock options, profit sharing, etc.) is “icing on the cake” and not necessary for the person to pay their normal living expenses
3. You don’t directly control the person’s paycheck (you both work for somebody else)

Not every situation will be ideal, but if you can’t handle requirements #1, and #2, maybe you should wait to hire until you can. There are people our there that will work hard without adequate compensation (either real or imagined), but your chances of hiring them are slim indeed.

3 Strikes and You’re Out

People that don’t perform well waste the time and energy of those that do. It’s really that simple. If you want the best possible performance from your team, each and every team member has to perform at their best: Team members with sub-par performance  have to “shape up or ship out” (even if they are friends). If you could just hire top performers to begin with, it seems like this would never be an issue, but hiring top talent every time is next to impossible and I’ve found that even top performers go through slumps. But you can identify people that don’t perform to expectations and it’s possible to pull people out of slumps. I’ll explain how I’ve done it below.

First, let’s define “best” a little more carefully: You’re the boss so you get to decide what is “best”. Maybe you want a shorted schedule (cutting features and quality to compensate). Maybe you want the best possible user-interface experience. Or maybe you want relatively bug-free code with a good, clean architecture. Whatever “best” is, it’s your choice and you need to decide and articulate your requirements and expectations clearly to your team.

Now let’s talk a little about “reasonable expectations”: Do you expect your team to work 60+ hours a week? In some cases this makes sense – a team working hard at “crunch time” to hit an external deadline, a tight knit group hoping to get venture capital funding or big bonus checks… 60+ hours weeks are OK, but, IMO, it’s unreasonable to expect people to do this week after week for years on end. Are there necessary tools are skills that are missing? There are two questions I ask myself: “What can I do to help?” and “Is what I’m asking for reasonable?”.

With all that background, now I’ll explain my “3 strikes” policy: People that work for me get fired after their 3rd written notice. In practice, this actually works out to 4 written notices because I usually don’t tell people out my “3 strikes” policy until what amounts to a written warning. Here is an example:

Boy, I sure am not happy to be writing this email, but tomorrow, unless you XXX, I am going to have to YYY. [...] These projects are all ultimately my responsibility and I can’t blame the lack of progress on external forces. [...]

You need to sit down and take a hard, honest look at yourself and your schedule and decide if you really have the time and willingness available for a project like this. If the answer is “yes”, then XXX.

Here at XXX, we have have a formal employment policy of 3 written notices, then you are fired. This email is a warning… please, please do not make me give you a written notice. This is not my idea of fun things to do. What I want is to have fun working with my friends, but it is not fun for me when I have to do all the work, write emails like this and/or receive angry calls from our customers.

Here are some concrete suggestions:

First and foremost, you need to spend several hours each day coding. On a project like this, you simply cannot procrastinate then get everything accomplished with one all night long programming marathon. Check out the section “The Three Hour Rule” in this article: http://www.w-uh.com/articles/030308-tyranny_of_email.html. Consider dedicating 2-3 blocks of 3 hours each day coding until you’ve worked yourself back into good graces.

XXX is your responsibility. I understand that YYY said he would do ZZZ, but you need to roll up your sleeves and do whatever is necessary to get this finished. Can you XXX? If you don’t do it, I will and like I said, it’s no fun for me when I have to do all the work.

There is no need to get stuck in the details of C#, XML, JavaScript, etc., when you can just pick up the phone and call or email me. If you haven’t figured it out yet, I am doing everything I can think of to make it easy to you to be successful. My guess is that you aren’t really putting in the time. However, if you are really getting stuck on programming details, that is just as bad when we both know you can can just pick up the phone and call me any time day or night.

These emails are tough to write and they take a lot of time, but I think it’s worth it. I didn’t put in all the gory details, but I think you can understand the idea that I’m trying to let the person know in as explicit terms as possible what it is that I want and that I’m willing to help them succeed. Firing somebody is a big deal and bringing somebody out of a slump is a really tough job. I think the “call me any time day or night” policy is important – often, challenges would not become major problems if the person had communicated.

Unfortunately, some people just can’t be saved. I’ve run into two situations that ended with people getting fired:

• The person is not capable of performing at a high enough level – some people really are just just not a good fit for your team. This is the toughest case because often the person is trying really hard. Now that you know the person better, if your company is big enough, maybe there is another position that better matches their capabilities.
• The person is not willing to do what it takes. This situation is not fun either, but by the time you get to the 4th written notice, it should be obvious to everybody that things aren’t going to work out. Usually, the person has made some clear choice to miss a deadline without letting me know in advance.

Firing somebody is tough, but it’s not the end of the world: Maybe your expectations are too high – if you’re a team leader, you’re probably a good bit above average – one way to learn is to fire and hire (a better way is to consult with others, but in small companies, this is not always possible). Or maybe the other person needs to do some soul searching. A nice, smart guy I hired once couldn’t perform. We went round and round about his lack of accomplishment (this was before I figured out the “3 strikes” policy). On a personal level, he was a great guy and I know he was having serious problems at home. I gave him as much slack as I could, but in the end I fired him. He actually called me a couple times afterwards. The first time was to let me know he was OK – he had found work and getting  fired had helped the home situation – it had been a wakeup call for his significant other. In the second call he told me that he was now leading a team and getting fired ended up being the best thing that could happened. Funny world.

Summary

Go ahead and hire your friends and have some fun at work. But:

1. Take care of all money issues up front
2. Make sure you’re doing your part – set expectations clearly and ask/discover what you can do to help
3. Carefully wield the stick of employment by using a clear policy of written notices

Question: Data Feed says 8660 of 8660 items inserted, but no Items (Active, Published or Inactive)

pjtpj Level 1 7/27/09

We have written a program that generates and uploads a tab delimited data feed file. This program was working fine for several months until around June 16. Starting around June 17, the program and the FTP upload has been working, but GoogleBase is not updating our GoogleBase items from the data feed file. We didn’t make any changes on our end and all required fields are in our data feed file. All our GoogleBase items eventually expired and were deleted last week.

Last Friday, I tried logging in to GoogleBase with my browser and manually uploading the data feed file. The data feed file was processed successfully, with a handful of the usual warnings and errors we always get (missing upc, encoding problems, missing discription). The missing UPC was a problem for all our products, but we only have 22 products with other errors. As before, I see the message “8622 of 8622 items inserted”, but I see no Items. Next, on Saturday, I tried adding a new Data Feed file. This time, the file got stuck “Processing” until today. Today, I finally got the error “Error: Internal server error while retrieving 9 # 0″. OK, so that is just some weird error.

Today, I changed our program to add UPCs where we have them and manually uploaded the data feed file again. I quickly got “8660 of 8660 items inserted – Processed on Jul 27, 2009 6:09 pm CST”, and the usual handful of warnings and errors (now I get some invalid UPC errors), but still I have no GoogleBase items.

I can manually add new Items and they become “Published and searchable”.

The Data Quality tab is basiclly empty (has information about the 1 item we created manually).

I tried creating a new GoogleBase account from scratch, but Google won’t let me create another account for the same web site (fair enough, I’m just trying to troubleshoot).

Any suggestions?

All answers

pjtpj Level 1 7/27/09

I just received a large batch of data feed related emails. The one associated with the most recent data feed file upload says:

No items inserted
Uploaded at: July 27, 2009 6:03:44 PM CST
Processed at: July 27, 2009 6:09:30 PM CST

For more details on the status of your data feed, go to http://www.google.com/base/showfeedsummary?fid=1336756

When I click on the link, I get the same “8660 of 8660 items inserted – Processed on Jul 27, 2009 6:09 pm CST” page I’ve already seen.

Celebird Top Contributor 7/27/09

what’s your website-url, google-base display-name and account-id?
has there ever been any disapproved messages seen within any account?

pjtpj Level 1 7/27/09

Website URL: http://www.mountainhighoutfitters.com
Display name: Mountain High Outfitters
Account ID: 2476386

No, we haven’t ever had any problems with disapproved items. There is one inactive/disapproved item right now – it’s the item with the title “Test Item 1″ I created on Friday to see if I could manually add an item.

Celebird Top Contributor 7/28/09

first, check the ‘view errors and suggestions’ page.

if there are no messages on the view-errors page –
you would need to contact google-base directly here:
http://base.google.com/support/bin/request.py?contact_type=error&ctx=direct

pjtpj Level 1 7/28/09

There is nothing the “view errors and suggestions” that would suggest why we have no items. As I mentioned earlier, we do have problems on 22 of the 8660 products, but there is nothing I would consider critical and nothing we weren’t seeing a month ago.

Today, I fixed a problem with custom attribute names (we were including spaces). This problem was always there, but I figured fixing it was worth a try. My upload just finished processing and it hasn’t seemed to help… still no items.

As you suggested, I filled out a support request. Thanks.

Kyle8 Level 1 7/28/09

Pjtpj / Celebird, I am experiencing exactly the same issue.  I upload our files and immediately get an email saying ‘No Items Inserted’ but when I click into my dashboard it says “x of x” (etc.) inserted successfully for all data feeds.  Under errors/suggestions it says I am missing the weight attribute but that the weight attribute is not required and will affect the feed. 

I have been experiencing this issue for over 2 weeks now.  I have been in contact with someone named Atanu at Google, & have sent well over 7 replies to him.  He has been somewhat responsive but each time just responds with a canned response saying “our engineers are working on resolving this issue. We sincerely apologize for the inconvenience.”  As I’m sure is the case with you, this has had a significant impact on our traffic and sales.  Has anyone else who is experiencing this problem found a workaround or a more solid response from Google?

eimaj Level 2 7/28/09

We had the same issue … identical, but our feed is Atom 1.0

We are making progress having successfully processed only 100 out of 30000+ products.  Little by little, we are adding more the the new feed and going from there.

Here’s what we did (for Atom 1.0 feed):
Having nothing to lose (our products expired as well), we deleted the feed file from Google Base dashboard
Made sure no products showed in www.google.com/base by authorid and by site:site.com search (count = 0)
Added a new feed file in the Google Base dashboard having a new unique file name different than all others used before
Created a new feed file having the exact content EXCEPT the //id was new, unique and different than all others used before – only having 2 products
Made sure those 2 products showed in www.google.com/base by authorid and by site:site.com search (count = 2)
Attempt at all 30000+ in a file failed … looked like we were back to square one
Attempt at 3000+ file failed … looked like we were still at square one
Attempt at 100 file succeeded!  We excluded products we did not have UPC or placeholder images, CDATA’ed the //entry/product_type and shortened the //entry/title in this file as well
We are at the point of progressively adding more products to the file trying to pinpoint the issue.

If anything is found, I will be more than happy to share the information. 

~Jamie

cheapercigars Level 1 7/28/09

ditto to pjtpj. I am facing the same issues with our feed just like pjtpj.
I have sent couple of emails to google.
still waiting for an answer.
lets hope its nothing serious.
- Mel

pjtpj Level 1 7/28/09

Thanks for all the replies. I haven’t heard anything back from Google, but I will post here when/if I figure anything out.

From the comments above, I suspect that the problem has something to do with the actual data in my data feed file. eimaj, maybe you can pinpoint the offending data via binary search: Divide the data in halves, then try to process each half separately. If one of the halves works, then divide the half that doesn’t work in halves (quarters of the original file) and try to process each new half separately and so on. This should at least be faster than trying 100 products at a time. Maybe something we added or updated around June 17 is causing the Google data feed file processing code to crash and/or reject all our items. Or maybe Google made some change to the code they use to process data feed files.

Here is the email I just got from today’s upload:

No items inserted
Uploaded at: July 28, 2009 7:37:05 AM CST
Processed at: July 28, 2009 8:10:40 AM CST

But, still, when I login to look at the data feed, I see “8660 of 8660 items inserted – Processed on Jul 28, 2009 8:10 am CST”.

This mismatch between the count of items inserted in the email and the web page is some kind of bug, but it doesn’t help point me to what is causing the problem. Just a little feedback from Google might go a long way…

Since others are reading this thread and having similar problems, I’m going to share some of my code and see if anybody else sees anything obviously wrong. Going with the theory that there is a problem in the data file, maybe there is a problem with the way I am encoding the file’s data fields. Here is the function I use to encode most data fields:

static char[] _quotedCharacters = new char[] { ‘\t’, ‘\r’, ‘\n’, ‘”‘ };

// GoogleBase specific rules
// See http://base.google.com/support/bin/answer.py?answer=25200&hl=en_US
public string TsvQuote(string s)
{
if (s.IndexOfAny(_quotedCharacters) >= 0)
s = “\”” + s.Replace(“\””, “\”\””) + “\””;

return s;
}

Here is the header line from my data feed file:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string

We use our own categories for product_type. The way the values are input into the system, the users can’t enter in non-printable characters, so I just do a simple search and replace for double quotes:

StringBuilder categoryText = new StringBuilder(category.Name);

while (category.ParentCategoryID != null)
{
category = Category.FindCachedCategory(category.ParentCategoryID.Value);
if (category == null)
{
categoryValid = false;
return “”;
}

categoryText.Insert(0, category.Name.Replace(“\””, “”) + ” > “);
}

return categoryText.ToString();

All the above code seemed reasonable at the time and was working OK until about June 16. OpenOffice has no problem parsing my file properly. I do have some HTML embedded into descriptions, and I’m not sure how GoogleBase treats that, but the HTML is not new.

pjtpj Level 1 7/28/09

Looking at my data feed file, I have a couple of ideas/thoughts:

1) I am worried about product_type. The GoogleBase specifications aren’t crystal clear… I coded up my best guess and it seemed to work, but maybe it really doesn’t. Here are some examples from my data feed file:

“Clothing > Men’s Clothing > Men’s Lifestyle Clothing > Men’s Lifestyle Pants”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $50 – $74.99″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $75 – $100″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”
“Gear > Camping/Hiking > Backpacks > Fast and Light Packs”,”Gear > Camping/Hiking > Backpacks > Weekend Packs”
“Accessories > Book Bags”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Activities > Adventure Travel > Travel Gear > Travel Packs”,”Gear > Camping/Hiking > Backpacks > Day Packs”
“Gear > Camping/Hiking > Backpacks > Day Packs”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Footwear > Men’s Footwear > Men’s Multisport Footwear”,”Activities > Climb > Climbing Shoes”,”Earth friendly Gear”
“Clearance > Winter Sale”,”Clothing > Kid’s Clothing > Kid’s Shirts”

As you can see, many of our products are in multiple categories / product_types and I believe we handle this correctly according to GoogleBase, but maybe somebody can see something I’m doing wrong.

2) I upload products with 0 quantity in stock. I know that these items are automatically deactivated, but that is what I want and it seems to be the most straightfoward way to make sure that products that go out of stock in our stores are removed/deactivated from GoogleBase.

cheapercigars Level 1 7/28/09

hello,
today in the morning I read that it may be a possibility that the file size is too big.
I read it in google help pages.
So, I divided my feed into two parts.
But no luck, the same thing happens. You can see the products being inserted but the email mentions that No Products Inserted. Also, when I checked the “data quality” tab, it does not show the new products.
So, if any of you get around it, please let me know on the thread. I am watching it.
Thanks
Mel

pjtpj Level 1 7/28/09

OK, so I think I am starting to narrow down the problem. Based on the idea from eimaj, I only kept the first 291 items in my data feed and eventually got back this email:

278 of 291 items inserted
Uploaded at: July 28, 2009 11:33:31 AM CST
Processed at: July 28, 2009 11:46:58 AM CST

And now a couple of hours later, I finally have 273 items “Published… searchable soon”. Between character encoding errors in description or quantity = 0, the missing items can all be accounted for. So, the problem seems to be somewhere in the content of my other 8369 items… Item # 292 has some Word HTML garbage – my current theory is that the source of the problem is with the HTML in some of the descriptions. I have figured out that some HTML is OK, because tables and bullet lists seem to make it through. One idea is that if any of the descriptions contain certain tags (<head>, <title>, <style>, <script>, etc.), then some final stage of data feed processing crashes and the entire feed ends up in the bit bucket.

cheapercigars Level 1 7/28/09

hello pjtpj
i understand that you had some “html code” in one of the line.
but my feed is strictly text, tab deliminated.
i did check it before posting this reply.
any more inputs will be appreciated.
I am just hoping that google might have fixed this issue (if there was something wrong with them to begin with)
Thanks
Mel

username6 Level 1 7/28/09

Did you try to submit data feed without custom attributes? We have exactly same problem. All items items just disappeared from active items tab.

cheapercigars Level 1 7/28/09

yes we did
but we really get high rankings when we have custom attributes.
I just heard from couple of good sources that FTP does not work.
This time I have tried to schedule the feed and let google fetch by itself.
lets see how this works.
i’ll keep it posted
Thanks
Mel

pjtpj Level 1 7/29/09

The 291 items that successfully processed yesterday had custom attributes. You can see the custom attribute I’m using in the header from my data feed file I posted above.

Kyle8 Level 1 7/29/09

Cheapercigars, if it’s the FTP that you’ve been hearing is down, do you know if the API is working? Anyone using the API just fine?

pjtpj Level 1 7/29/09

This morning, I created and uploaded a data feed file that avoided any product that had <head <title <style <script or <meta in its description. That did not work. Now I have taken that data feed file and broken it into 4 parts. I am processing all 4 parts right now. Maybe 1 or more part will succeed and help me pinpoint the source of the problem.

pjtpj Level 1 7/30/09

Yesterday, I worked it down to a data feed file with only 9 items that causes the problem. I have tried 3 different possible guesses as to what in the file is causing the problem, but I haven’t figured it out yet.

This morning, I got a fairly generic email response from GoogleBase Support:

Thank you for your email. I have reviewed your Google Base account and the
reason few items your file are failing is because your feed contains
unsupported characters. For example, certain white space characters, such
as tabs, may not process correctly and will result in a feed failure.

To correct this error, please remove all unsupported characters like HTML
tags etc from your feed and upload your file again.

Please let me know if you have any further questions or concerns.

Unfortunately, this response was not helpful. I already knew about the “few items” and, as mentioned here, I have submitted multiple data feed files that don’t have any of these issues, but still fail to update my GoogleBase items.

pjtpj Level 1 7/30/09

I have finally narrowed this problem down to just 1 product/item. Here is a copy of the data feed file that will not process successfully:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
“<ul>
<li>A multipurpose rescue/rappel rope.</li>
<li>100% nylon low stretch static.</li>
<li>Uniform sheath coverage for durability and gear compatibility .</li>
<li>Ideal for heavy duty alpine, big wall, and expedition fixing and haul lines.</li>
<li>Great for mountain rescue and industrial rope access.</li>
<li>Certified to NFPA 1983, 2006 edition for Life Safety Rope forlight use.</li>
</ul>” 9995 http://www.mountainhighoutfitters.com/Products/SuperStatic+7%252f16%2522+-+300ft.html 255.00 “Sterling Rope SuperStatic 7/16″” – 300ft” Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-1.teztech.com/Products/7662.jpg 8.00 pounds 1 new

My current theory is that it is the URL that is tripping it up. This URL is valid (works in IE, FF and Chrome), but it does contain URL encoded characters ( / and ” ) that could cause problems if unescaped. The other interesting thing about this product is that it was added to our database 14 May 2007, so it was working OK at some point (we were able to upload all our products to GoogleBase successfully in the past).

Celebird Top Contributor 7/30/09

i would not include any html.

pjtpj Level 1 7/31/09

I made a mistake in my last post. Actually, that data feed file in that post, even with the HTML and complex URL will process OK. Here is the file that will not process successfully:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
The Nose in a day, dogging your project for the painful 20th day, and everything in between, this 10.4mm rope handles all ends of the spectrum. Lightweight, it will never slow you down for a second and will always return day after day to handle the abuse your projects dish out. 39705 http://www.mountainhighoutfitters.com/Products/Marathon+Sport+70m.html 224.00 Sterling Rope Marathon Sport 70m – 70m Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-2.teztech.com/Products/7661.jpg 7.00 pounds 2 new 70m

There are no double quotes around the description because it shouldn’t need them.

I am going to submit this file again right now just to be sure.

PJ

Celebird Top Contributor 7/31/09

not including html was more a policy recommendation not about formatting –
but html can cause formatting issues as well so isn’t recommend at all.

quotes cannot be determined to be “needed” or “not needed”
without knowing what the quoted-fields setting is for the feed –
each feed can be set differently and effect quotation requirements.

without a url link to the file and knowing the settings for the registered-feed
the copy-and-paste does not reflect what google-base is doing with the file –
the same is true for issues such as encoding.

also, currently, only one product_type value is supported in tab-delimited feeds.

pjtpj Level 1 7/31/09

I haven’t seen anything that says if HTML is allowed or not. I have noticed that HTML seems to be stripped from my data feed, but is allowed for manually submitted items. I have not noticed that HTML prevents items from getting posted (or, as in my case, causes some internal problem with data feed file processing).

Tab and comma delimited files existed long before Google decided to allow their use for data feed files. My understanding has always been that, in these types of files, a field “needs” to have quotes around it if it contains the delimiter character or line feed. Here is what Google has to say about this subject: http://base.google.com/support/bin/answer.py?hl=en_US&answer=25200. It is not explained how the use of quotes interacts with the  “Use Quoted Fields” setting, but I always set this to “Yes” under the assumption that “Yes” allows the use of quotes around fields (but does not require it). I have many successful runs that use the code I posted above, so I think, at least to some degree, GoogleBase also understands that fields within a tab delimited file may or may not have quotes around them, depending on if they are “needed” or not.

I am still working on this (and just hoping that maybe somebody else sees something obvious)… if I find out for sure that the data feed file I’m working on now with just a single product is causing the problem, then I will post the file for downloading. Here are the settings I use for all my data feeds:
Feed format: googlebase, File format: text, Encoding: autodetect (detected as UTF-8), Delimiter: tab, Use Quoted Fields: Yes

Where did you read that only one product_type value is supported in tab-delimited feeds? This page seems to suggest that you can have more than one: http://base.google.com/support/bin/answer.py?hl=en&answer=66818. Looking at the raw data feed file and some items that have gotten through, I figured out that I probably need to quote the comma separated product_type values individually rather than trying to quote the entire field (right now, all my product_type values end up with double-quotes around them). But this is a minor problem and I am failure sure it’s not the cause of whatever is keeping my big file from getting processed.

Celebird Top Contributor 7/31/09

i wouldn’t include any html in feeds.

google-base feed quotes exist within the context of
feed settings apart from any tab-delimited standards.

i believe use-quoted-fields set to “no” has always meant
to turn-off the feature so quotations will be seen as part
of the value — set to “yes” the feature is operational so
quotations will be used as a field delimiter, not a value.

also, length is a google-base defined attribute so
probably shouldn’t be redefined as a custom-attribute.

what exactly does the ‘view errors and suggestions’ page display?

understood about the help information and the product_type attribute — see also:
http://www.google.com/support/forum/p/base/thread?tid=476189d91e13bdaa&hl=en
http://www.google.com/support/forum/p/base/thread?tid=77a9e517542cee97&hl=en

pjtpj Level 1 7/31/09

I’ve uploaded the same file (with only one product) twice in a row now with the same problem occurring, so I think I have it down to a repeatable test case. Here is a link to the file: http://teztech.com/files/GoogleBaseProducts4F1.txt. I uploaded the file in binary mode to make sure it was the same as what I am uploading to Google. After I upload the file and it is processed, here is what GoogleBase says when I login:

1 of 1 items inserted – Processed on Jul 31, 2009 7:20 am CST
Detected encoding: US-ASCII
Detected delimiter: Tab (\t)
Detected file format: Text
Feed Level Messages
Warning: Missing recommended attribute: upc
While items missing recommended attributes will process successfully, we recommend including relevant attributes if they are available.

And here is what the email from GoogleBase says:

No items inserted
Uploaded at: July 31, 2009 6:33:37 AM CST
Processed at: July 31, 2009 7:20:04 AM CST

The “No items inserted” email is how I determine that “the problem” has occurred. I have also noticed that it takes a relatively long time to process these files. I’m sure they go into a queue and the processing time depends on how busy the GoogleBase data feed servers are, etc., but I’ve been processing these files manually for 5 days in a row and I think an hour to process a 1 product data feed file is longer than usual.

The GoogleBase data feed documentation explains how to create tab delimited files by hand in Excel and OpenOffice Calc, so I would assume that, at least to some degree, the GoogleBase programmers had the output of these programs in mind when they created their import code.

Celebird, I agree that “length” probably shouldn’t be a custom attribute. Thanks for the second pair of eyes. I’m going to try making some changes to length and see what happens. Interestingly, the length in my data feed is the length of the product (a rope in this case) and not the length of the shipped package. I wonder which one Google means by “length”? If they can use length to calculate shipping charges, then it is the length of the shipped package. I’ll try and research this.

Thanks for the pointers to the product_type threads. I just received an email with “Our engineers are aware of this problem and are working to resolve it.” on my issue, so, unfortunately, this seems to be some kind of standard response.

pjtpj Level 1 7/31/09

1 person says this answers the question:

Changing the name of my length field from c:length:string to c:product_length:string allowed the file with 1 product to be successfully processed. Now let’s see if I can process a real data feed file.

ajj Level 1 7/31/09

Hi, in case the products are still not showing try this:  I see you’re using multiple categories – hence the reason for my recommendation:

1. On the google base dashboard, click My items tab.  Then click Data feeds.  Next click the edit link adjacent to you data feed file so you can set or verify the base configuration.

2. You should be in the “Edit Data Feed” screen at this stage.  Verify the following (or adjust to have them)

File format:                     Text
Encoding:                       Autodetect
Delimiter:                        Tab
Use Quoted Fields:          No

Note, the last item should a No and NOT Yes in your case.  Then resubmit.  Hope this helps

pjtpj Level 1 10:51 AM

Here is the source of my problem: GoogleBase processing crashes when I have a custom attribute named “c:length:string” and a value of “70m” (I assume other non-empty values cause problems as well). I changed my custom attribute name to “c:product_length:string” and now, the email I just got back from Google says: “8584 of 8606 items inserted”.

Some might say that this was an obvious problem, but our Internet store allows our end users to create their own attribute groups (color, size, style, etc.). When we create a data feed file, there is some code that matches our attribute groups to standard Google attributes (color, size). If there is not a match, then we automatically create a custom attribute for GoogleBase. In our case, “length” in our data base is the product length, not the shipping length, so using a custom attribute actually make sense.

In addition, I ran into other issues that caused a simple problem like this to take an entire week to work out:

- Even though GoogleBase processing was failing, the data feed message on the web site said “8660 of 8660 items inserted”. The GoogleBase data feed processing code was crashing on my data feed file, but apparently GoogleBase doesn’t code in place to trap these types of errors and provide useful diagnostics to outsiders like me.
- In my 8660 product data feed file, only a few products caused the problem and there was no easy way to narrow down the source of the problem. I had to manually divide my file into parts, submit the parts manually, then wait on the “success” emails to arrive, then rinse and repeat.
- There was nobody at GoogleBase support with the willingness or ability (not sure which) to help me troubleshoot this issue.
- Documentation for data feeds is just vague and wrong enough to keep you guessing. The biggest documentation issue I see right now is with product_type and how it interacts with the Use Quoted Fields setting.

If you run into a similar problem, the first step is to make sure you are getting all the data feed emails. Only the email said “No items inserted”. Next, you must isolate the data that is causing the problem. I assume my problem with a custom attribute named “c:length:string” is pretty specific, but the technique I ended up using to isolate it is general (if slow and painful).

Celebird Top Contributor 11:33 AM

these are many of the same similar reasons why it seemed a good idea
to write a feed validation checker with an ever expanding set of checks –
http://celebird.googlepages.com/ckbase.htm

also, 70m is proabably an invalid standard-length value — a valid length would be 70 m

“standard” attributes can be google-defined attributes but
are in the context of an item-type; not all defined-attributes
are allowed or are valid within the context of an item-type –
an obvious example is location; item-type requirements are
well-defined and i believe must be taken into account if any
private-code is creating the final set of attributes and values.

google-base processing has always been more simple counts
and basic formatting rather than robust validation; this is most
evident with respect to encoding, quoted-attributes and other
per-feed features — i believe this is one reason why the feed’s
suggestions feature was introduced.

pjtpj Level 1 12:10 PM

GoogleBase actually does an OK job of identifying some problems (with the suggestions feature). If GoogleBase would catch and report errors like mine that cause a complete crash (I assume) and include the data feed file line number at the time of the crash, that would be a big help.

Celebird, I appreciate the effort on both the checker and on this thread -it was your comment that led me to the solution. I didn’t know about your checker. I just now submitted my test file (http://teztech.com/files/GoogleBaseProducts4F1.txt) and it passed with warnings. You could add a test to check for custom attributes with the names of standard attributes. That test file also has lots of quotes within the product_type field… from my own tests and comments from others, it looks like if you want to use multiple categories, then you have to set “Use Quoted Fields” to “No” (which then won’t allow newline characters in the description because you can’t use quotes to delimit a field). You could add a check for this as well, but actually, I hope GoogleBase cleans up their act on defining and parsing this field. XML is such a waste, I really don’t want to switch file formats just for this, but I also admit that text files have their own problems.

Yes, it’s possible that “70 m” vs. “70m” could have fixed the problem as well. On my side, “length” is a attribute group defined by my customers and the values entered are free text strings (which makes sense for rock climbing ropes that come in 50m, 60m and 70m), so the safest course for me was to rename the field. At some point, I plan to add support for dimensional shipping. When I do that,I’ll go back and figure out what Google means by “length” and rework things accordingly.

Recently, I upgraded my development system from OpenOffice 2.x to OpenOffice 3.1. I can’t remember now the main reason I upgraded, but I was looking forward to being able to add the ability to convert docx to pdf (Office Open XML support was added in OpenOffice 3.x). I figured the upgrade might require some minor changes to my document conversion code, but it turned out not to be so simple.

The first problem was that OpenOffice no longer showed up under “Add Reference” in Visual Studio. After some Google searches, I finally stumbled across a solution. OK, wow, digging though install cab files for DLLs, hacking the registry and manipulating the PATH is ugly, but it worked – at least once. Almost as soon as my test program ran successfully, Visual Studio locked up. The only way to get control back was to kill the Visual Studio process, then restart it. Every time I opened the solution file, after just a few seconds, Visual Studio would lock up again. Removing every trace of the OpenOffice DLL references would prevent the lockups, but without OpenOffice, I could not convert documents. I eventually figured out that the problem was that the library project that used OpenOffice was also included in my web site project. Because the web site uses the library, too, the OpenOffice reference is automatically copied to the web site project. Normally, this would be no big deal – just a couple of small DLL files in the web site bin folder that would not be used – but in this case, Visual Studio could not successfully build the web site project as long as the OpenOffice DLLs were referenced.

Now that using OpenOffice was no longer as simple as “Add Reference”, it was time to take a look at a way to decouple my application from the OpenOffice DLLs. The API my application needs is pretty simple: Given the bytes of a file in Doc (or .docx) format, return the bytes of the file in Pdf format. Also, there are security risks whenever you open a file in a complicated piece of software like OpenOffice. These two facts pointed me to turning the Pdf conversion into a web service. The client application and web service can be partitioned across computers, so, if I wanted to, I could run the Pdf conversion service on a virtual machine with its own install of OpenOffice and tight security and resource control. The web service API didn’t need fancy XML: Clients could post the Doc file via a standard HTTP MIME encoded form. The result would be a the PDF byte stream returned via HTTP. This would make the service easy to develop, test and use from clients on many platforms.

I typically develop my large web based applications in C# and Asp.Net. However, for small services like document conversion, I often use PHP. Simple PHP applications can be just a file or two and I can move a PHP based service over to any web server that supports PHP. So, my first attempt at a document conversion service was written in PHP and used the OpenOffice COM API. Using my earlier OpenOffice API code as a guide and some PHP sample code I found on the web, I was able to quickly create a command line PHP application that could perform Pdf conversions. With this proof of concept working, I coded up the PHP web service. But the web service wouldn’t work. I could see in Task Manager that the OpenOffice executables were launched, but any code running under the web server that tried to access the OpenOffice COM objects would freeze up.

I spent a lot of time trying different security settings for various files and DCOM. I even tried creating a dedicated user and configuring DCOM to run OpenOffice as the dedicated user when the OpenOffice COM objects were activated. I changed Windows event log settings to log all security violations. Nothing would work and I couldn’t see anything useful in Event Viewer. I tried coding up a simple Asp.Net application and got the same results under the .Net OpenOffice API. Under both PHP/COM and C#/.Net, OpenOffice could be automated by a command line program but would not run under IIS. Right now, I develop on Vista and deploy to Windows Server 2008, so one might think the problem was trying to use IIS 7, but OpenOffice 2 didn’t have similar problems.

Finally, I gave up trying to run OpenOffice directly under IIS. The problem was either some type of security issue or that OpenOffice was trying to pop up registration dialogs. In Windows, the simple way to run a program under a given user account is to run the program as a Windows Service. It turns out that it is pretty easy to create a Windows Service in C#. The basic idea I came up is was:

1. Install OpenOffice
2. Create a Windows user account dedicated to Pdf conversion.
3. Login as the Pdf conversion user and make sure that OpenOffice opens without prompting for registration, crash feedback, etc. (I suspect that the reason OpenOffice 3 won’t run properly under IIS is that it is prompting the user to register).
4. Create a Windows Service with a simple API to convert documents
5. Install the Windows Service and configure it to run as Pdf conversion user
6. Access the Windows Service from my document conversion Web Service

Here is how the API calls flow: Application -> Web Service -> Windows Service

As I mentioned earlier, the API between the application and the web service was standard a HTTP Post. Between the Web Service and the Windows Service, I originally planned to use a simple, proprietary TCP/IP protocol. As long as you aren’t try to get too fancy coding thread pools and asynchronous I/O, at least in C/C++, it’s pretty easy to open a TCP socket and communicate between client and server with a line oriented protocol (like HTTP, POP3, SMTP, etc.). But, since both the Web Service and the Windows Service were going to run on Windows (at least for now), I decided to take a look at .Net Remoting. The .Net remoting solution was easy to code and use so I ended up sticking with it.

Below is some code to show you how everything fits together:

The C# interface IConverter is compiled into a DLL. The DLL is referenced by the client (Web Service) and server (Windows Service).

    public class ConverterEndpoints
    {
        public static int    Port = 7047;
        public static string Uri  = "DocumentToPdf";
    }

    public interface IConverter
    {
        void Convert(string srcDocument, string destPdf);
    }

The class ConverterEndpoints holds a couple of constants that the client and server use to setup the TCP port and service name. The code below demonstrates how ConverterEndpoints is used in the Windows Service. I’ve included code for the entire class so you can see how easy it is to program a Windows Service and access the Windows Event Log in C# – no 3rd party libraries are required.

The original version of this interface passed the files as simple byte[] variables. Theoretically, the files never need to be written to disk (input and output is via HTTP). But, I found that I could easily pass small byte[] variables across .Net remoting but large byte[] variables failed. Since the Web Service and the Windows Service run on the same machine and the file has to be written to disk for OpenOffice to do the conversion, I settled for creating temporary files and passing the names of the files between the Web Service and Windows Service.

   class WindowsService : ServiceBase
    {
        protected EventLog _log;
        protected TcpChannel _channel;

        public WindowsService()
        {
            ServiceName = "Teztech Document Conversion";
            EventLog.Source = "Teztech Document Conversion";
            EventLog.Log = "Application";
            
            // These Flags set whether or not to handle that specific
            //  type of event. Set to true if you need it, false otherwise.
            CanHandlePowerEvent = false;
            CanHandleSessionChangeEvent = false;
            CanPauseAndContinue = true;
            CanShutdown = true;
            CanStop = true;

            if (!EventLog.SourceExists("Teztech Document Conversion"))
                EventLog.CreateEventSource("Teztech Document Conversion", "Application");

            _log = new EventLog();
            _log.Source = "Teztech Document Conversion";

            _channel = new TcpChannel(ConvertInterface.ConverterEndpoints.Port);
            ChannelServices.RegisterChannel(_channel, false);

            RemotingConfiguration.RegisterWellKnownServiceType(typeof(DocumentToPdf), ConvertInterface.ConverterEndpoints.Uri, WellKnownObjectMode.Singleton);
        }

        static void Main()
        {
            ServiceBase.Run(new WindowsService());
        }

        /// <summary>
        /// Dispose of objects that need it here.
        /// </summary>
        /// <param name="disposing">Whether or not disposing is going on.</param>
        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);
        }

        protected override void OnStart(string[] args)
        {
            _log.WriteEntry("Service Starting");

            _channel.StartListening(null);

            base.OnStart(args);

            _log.WriteEntry("Service Running");
        }

        protected override void OnStop()
        {
            _log.WriteEntry("Service Stopping");

            _channel.StopListening(null);

            base.OnStop();
        }

        protected override void OnPause()
        {
            _channel.StopListening(null);
            base.OnPause();
        }

        protected override void OnContinue()
        {
            _channel.StartListening(null);
            base.OnContinue();
        }
    }

Below is the code behind file for the Web Service client. As you can see, .Net makes this type of Web Service easy to create.

public partial class Pdf : System.Web.UI.Page
{
    protected TcpChannel _channel;

    protected void Page_Load(object sender, EventArgs e)
    {
        try
        {
            if (Request["Username"] == null || Request["Password"] == null)
                throw new ApplicationException("Username and Password are required");

            HttpPostedFile postedFile = Request.Files["InputFile"];
            if (postedFile == null || postedFile.FileName == null || postedFile.FileName == "")
                throw new ApplicationException("InputFile was not supplied or file name is empty.");

            string inputFileExtension = Path.GetExtension(postedFile.FileName);
            if (inputFileExtension.Length < 2)
                throw new ApplicationException("InputFile does not have a valid file name extension.");
            inputFileExtension = inputFileExtension.Substring(1);

            string outputFileName = Request["OutputFileName"];
            if (outputFileName == null || outputFileName == "")
                outputFileName = Path.GetFileNameWithoutExtension(postedFile.FileName) + ".pdf";

            DbRequest.CheckAuthorization(Request["Username"], Request["Password"]);

            if (_channel == null)
            {
                Dictionary<string, string> channelProperties = new Dictionary<string,string>();
                channelProperties["name"] = "";
                _channel = new TcpChannel(channelProperties, null, null);
                ChannelServices.RegisterChannel(_channel, false);
            }

            string url = string.Format("tcp://localhost:{0}/{1}", ConvertInterface.ConverterEndpoints.Port, ConvertInterface.ConverterEndpoints.Uri);

            ConvertInterface.IConverter converter = (ConvertInterface.IConverter)Activator.GetObject(typeof(ConvertInterface.IConverter), url);

            string path = MapPath(".");
            path = Path.Combine(Path.GetDirectoryName(path), "Temp");

            using (TempFileCollection tempFiles = new TempFileCollection(path))
            {
                string inputFile  = tempFiles.AddExtension(inputFileExtension);
                string outputFile = tempFiles.AddExtension("pdf");

                postedFile.SaveAs(inputFile);
                converter.Convert(inputFile, outputFile);

                Response.Clear();
                Response.ContentType = "application/pdf";
                Response.AddHeader("content-disposition", string.Format("inline; filename={0}.pdf", "sample"));

                byte[] pdfBytes = File.ReadAllBytes(outputFile);

                Response.OutputStream.Write(pdfBytes, 0, pdfBytes.Length);
            }
        }        
        catch (Exception ex)
        {
            Response.Clear();
            Response.Write(string.Format("<HTML><HEAD></HEAD><BODY><H1>Conversion Error</H1><P>{0}</P></BODY></HTML>", ex.Message));
            Response.StatusCode = 500;
            Response.StatusDescription = ex.Message;
        }
    }

For completeness, I have included the code for the actual Web Service’s .aspx file below. As you can see, it’s just standard .aspx. boilerplate – all HTTP output is generated in the code behind file’s Page_Load event (which will be invoked when the Web Service client posts to the page). This is quick and dirty, but it works.

<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Pdf.aspx.cs" Inherits="Pdf" EnableViewState="false" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    
    </div>
    </form>
</body>
</html>

Here And here is the code for a little PHP application I used to test my conversion Web Service:

<?php $ErrorMessage = ''; ?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>
File Conversion Server Demo
</title>
</head>
<body>

<h1>File Conversion Server Demo</h1>

<form name="Demo" method="POST" enctype="multipart/form-data" action="/demo.pdf">

    <?php if($ErrorMessage): ?>
    <p><font size="5" color="#008000"><?php echo $ErrorMessage; ?></font></p>
    <?php endif; ?>
    
    <table>
        <tr>
            <td><b>Username:</b></td>
            <td><input name="Username" value=""></input></td>
        </tr>
        <tr>
            <td><b>Password:</b></td>
            <td><input name="Password" value=""></input></td>
        </tr>
        <tr>
            <td><b>Input File:</b></td>
            <td><input type="file" name="InputFile"></input> </td>
        </tr>
    </table>
    
    <input type="submit" value="Submit"></input> 
    
</html>

If you look the PHP form’s action tag, you’ll see I used /demo.pdf as the action. Using the very nice IIS7 URL Rewrite Module, I map all *.pdf URLs to my conversion Web Service. OK, this might be a little overkill, but this way the URLs look nice and all end in .pdf (which is appropriate).

The final piece of the puzzle is a small wrapper class I created to access the conversion Web Service from within my C# application:

    public class ConvertClient
    {
        public ConvertClient(string convertHost, string username, string password)
        {
            _ConvertHost = convertHost;
            _Username    = username;
            _Password    = password;
        }

        protected string _ConvertHost;
        public string ConvertHost { get { return _ConvertHost; } set { _ConvertHost = value; } }

        protected string _Username;
        public string Username { get { return _Username; } set { _Username = value; } }

        protected string _Password;
        public string Password { get { return _Password; } set { _Password = value; } }

        protected string _Response;
        public string Response { get { return _Response; } set { _Response = value; } }

        protected int _MaxConvertAttempts = 2;
        public int MaxConvertAttempts { get { return _MaxConvertAttempts; } set { _MaxConvertAttempts = value; } }

        protected string _ResponseCode;
        public string ResponseCode { get { return _ResponseCode; } set { _ResponseCode = value; } }

        protected static string _szBoundary    = "SEPARATORSTRINGTEZTECHDOTCOM1";
        protected static string _szBoundary2   = "\r\n--SEPARATORSTRINGTEZTECHDOTCOM1\r\n";
        protected static string _szBoundary3   = "\r\n--SEPARATORSTRINGTEZTECHDOTCOM1--";
        protected static string _szFileSizeHdr = "Content-Disposition: form-data; name=\"MAX_FILE_SIZE\"\r\n\r\n";
        protected static string _szFileHdrFmt  = "Content-Disposition: form-data; name=\"InputFile\"; filename=\"{0}\"\r\nContent-Type: application/octet-stream\r\n\r\n";

        public byte[] ConvertToPdf(byte[] inputFileBytes, string inputFileName, string outputFileName)
        {
            string url = string.Format("http://{0}/{1}?Username={2}&Password={3}", ConvertHost, outputFileName, Username, Password);

            // Calculate upload data size

            string szFileSizeData = string.Format("{0}", inputFileBytes.Length + 50000);
            string szFileHdr      = string.Format(_szFileHdrFmt, inputFileName);

            ASCIIEncoding ascii = new ASCIIEncoding(); // At this time, file names must be ascii
            List<byte> header = new List<byte>(); 
            List<byte> footer = new List<byte>(); 

            header.AddRange(ascii.GetBytes(_szBoundary2));      // MAX_FILE_SIZE field
            header.AddRange(ascii.GetBytes(_szFileSizeHdr));
            header.AddRange(ascii.GetBytes(szFileSizeData));
            header.AddRange(ascii.GetBytes(_szBoundary2));      // userfile field
            header.AddRange(ascii.GetBytes(szFileHdr));

            footer.AddRange(ascii.GetBytes(_szBoundary3));
            
            int cbContent = header.Count + inputFileBytes.Length + footer.Count;

            int attempts = 0;

            while (true)
            {
                try
                {
                    attempts++;

                    HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
                    webRequest.Method        = "POST";
                    webRequest.ContentType   = string.Format("multipart/form-data; boundary={0}\r\n", _szBoundary);
                    webRequest.ContentLength = cbContent;

                    using (Stream request = webRequest.GetRequestStream())
                    {
                        request.Write(header.ToArray(), 0, header.Count);
                        request.Write(inputFileBytes, 0, inputFileBytes.Length);
                        request.Write(footer.ToArray(), 0, footer.Count);
                    }

                    using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
                    {
                        using (BinaryReader reader = new BinaryReader(webResponse.GetResponseStream()))
                        {
                            byte[] pdfBytes = reader.ReadBytes((int)webResponse.ContentLength);
                            return pdfBytes;
                        }
                    }
                }
                catch
                {
                    if (attempts >= MaxConvertAttempts)
                        throw;
                }
            }
        }
    }

I didn’t see an easy way to build a MIME document in .Net, so I’m building up the MIME document (used in the HTTP Post) from scratch.

To debug the Windows Service, as shown in the code above, I added code to log to the Windows Event Viewer. In this case, I started with working document conversion code, so I didn’t end up needing to run the service under a debugger. But, one of the nice things about services running as a dedicated process (unlike Apache modules, ISAPI modules and Control Panel applets) is that, with just a simple change (adding a Main function), you can run the service as a regular command line application and attach the Visual Studio debugger to the running process.

To troubleshoot the Web Service, my first step was to disable “friendly errors”. With IIS7, you have to do it on both the web browser client and in IIS7. Debugging is just a matter of attaching the IIS7 process in Visual Studio (as usual for any web application).

All our production Windows servers use the 64 bit edition of Windows 2008 Server. Right now, I happen to do most of my development on a 32 bit edition of Windows Vista. Both platforms run IIS7 and usually, the 32 vs. 64 bit doesn’t cause any problems. However, the Windows version of OpenOffice only comes in a 32 bit version. This version runs fine on 64 bit Windows, but when programming OpenOffice via .Net, you need a 32 bit application, so you have to set the Target Platform in the Project Build Properties to X86.

http://www.lotterygold.com/ This was an interesting and very large project that recently went live. The application we created for this site is general purpose and can support many types of e-commerce. The backend of this site is very comprehensive and highly automated. Due to its sensitive nature, the there are multiple layers of security and a dedicated audit trail facility.

http://www.mountainhighoutfitters.com/ This site is based on our very comprehensive and sophisticated online store engine. Our client’s goal is to compete head on with the likes of rei.com (who probably have spent hundreds of thousands if not millions of dollars on their online store). We have a very full featured marketing capabilities, integrated product catalog management (with lots of JavaScript to make it friendly and easy to use), integrated order fulfillment and automated integration with in-house store inventory. Since Google is the source of so much traffic (both paid and non-paid), we support every service that Google offers: Google Base (a.k.a. Google Products and Froogle), Google Site Maps, Google Adwords, Google Checkout and Google Analytics. We have some interesting payment solutions for vendors that want to sell to non-traditional markets (overseas). All our web applications support our standard skin and content management system (see below), so just about any look and feel can be created without any programming or database changes. There is a lot of power and flexibility under the hood that is probably best demonstrated in a meeting.

http://www.hudhomeschattanooga.com/ This is a standard ListingsTech site to market HUD homes in the metro Chattanooga, TN area. One interesting thing about this project is that we allow visitors to search across subsets of two different listings databases. We needed to do this because the databases contain all HUD listings for Georgia and Tennessee, but the site is only for Chattanooga. Another interesting idea here is the general approach of keeping the site as simple as possible – we are trying to get visitors to the information or service they want in as few clicks as possible… Want to see listings for Walker County? Click the search term on the left. Need to get prequalified for a loan? Click the link on the top of the page and fill out the form. Need more information about a listing? Click the “Contact Us for More Info” link and type your question.