• Why Linq beats SQL
• IQToolkit (for building Linq data source of any type)
• Linq to IMAP (looks very useful and a very cool demonstration of the possibilities)

The G1 was almost good enough, but I heard it was slow and slow computers drive me crazy. Finally, my Droid has arrived. I’m not thrilled that Java is the recommended programming language, but at least you don’t have to program in Objective C. I’m on day 2 with my Droid and I’m really just getting started. The first challenge I ran into was moving my contacts from Thunderbird and my old Casio “Boulder” phone. With my old Casio phone, I used Verizon’s free Backup Assistant to keep my contacts backed up. This application is actually a nice touch by Verizon and they should publicize it better. After your first backup, you can login to their site and see your contacts. Last time I switched phones, I just installed Backup Assistant on my new phone and downloaded my contacts from my online backup. Unfortunately, Verizon has not yet ported Backup Assistant to the Android platform. They do, however, allow you to login to their web site and download all contacts as a CSV files. I though for sure there would be an easy way to get contacts in a CSV file onto my Droid, but that did not turn out to be the case.

My first task was to export another CSV from Thunderbird and consolidate my cell phone and email contacts. Luckily, this wasn’t too bad because I had done it last time I switched phones. I used OpenOffice’s Calc program to open both CSV file, manually consolidate them, then export the resulting consolidated contact list as a CSV file.

To get my consolidated CSV onto the Droid, first, I tried copying the CSV file onto the Droid and using the phone’s import contact function. With a USB cable,  getting the CSV file onto the phone was easy enough, but the Droid import function only understands vCards, so the CSV file was useless for the Droid import function.

The Droid will automatically sync with Gmail Contacts and I knew Gamail had a CSV import function, so I tried uploading my CSV file into Gmail Contacts. I could get some of the CSV uploaded, but, after many tries, I could not get Email addresses and mobile phone numbers properly imported. These are the most important data items, so IMO, Gmail’s CSV import is almost useless.

Next I tried to use an online service to convert my CSV to vCard, then import the resulting vCard file via the Droid contact import function. This sort of worked, but contacts with multiple phone numbers had all phone numbers set to be “home” and all email addresses were set as “other” (rather than “work” or “home”). I got the bright idea to import the vCard file into Gmail contacts, but this produced the same problems as trying to import directly on the Droid.

Thinking that getting contacts into Gmail must be a common problem, I ran a bunch more Google searches looking for something that would do the conversion I needed, but I found nothing.

So, I decided to do what only a programmer can do… write a program to get my contacts from the CSV onto my Droid. Actually, I did this indirectly. I used Google’s .Net data API to upload my CSV into Gmail Contacts, then the Droid’s built-in Gmail sync function automatically puts the contacts onto my Droid. I did it this way because a) I’ve worked the Google .Net data API before and found it easy to use and b) I have a good CSV library I’ve found and used on other projects.

For those interested, I have copied the full source code below:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

using Google.GData.Contacts;
using Google.GData.Client;
using Google.GData.Extensions;
using Google.Contacts;

using LumenWorks.Framework.IO.Csv;

namespace GoogleContactsUpload
{
    class Program
    {
        static void Main(string[] args)
        {
            string username = "user1@gmail.com";
            string password = "password1";

            Uri feedUri = new Uri(ContactsQuery.CreateContactsUri(username));

            RequestSettings rs = new RequestSettings("teztech-GoogleContactsUpload-1", username, password);
            ContactsRequest cr = new ContactsRequest(rs);

            using (StreamReader contactsReader = new StreamReader(@"C:\Users\pj\Documents\Databases\ContactsMerged.csv"))
            {
                using (CsvReader contactsCsv = new LumenWorks.Framework.IO.Csv.CsvReader(contactsReader, true, ',', '"', '"', '\0', true))
                {
                    while (contactsCsv.ReadNextRecord())
                    {
                        Contact contact = new Contact();
                        contact.Title = string.Format("{0} {1}", contactsCsv["First Name"], contactsCsv["Last Name"]);

                        if (contactsCsv["Email Address"] != "")
                        {
                            EMail workEmail = new EMail(contactsCsv["Email Address"]);
                            workEmail.Primary = true;
                            workEmail.Rel = ContactsRelationships.IsWork;
                            contact.Emails.Add(workEmail);
                        }

                        if (contactsCsv["Home Email"] != "")
                        {
                            EMail homeEMail = new EMail(contactsCsv["Home Email"]);
                            homeEMail.Rel = ContactsRelationships.IsHome;
                            contact.Emails.Add(homeEMail);
                        }

                        if (contactsCsv["Work Phone"] != "")
                        {
                            PhoneNumber workPhone = new PhoneNumber(contactsCsv["Work Phone"]);
                            workPhone.Rel = ContactsRelationships.IsWork;
                            contact.Phonenumbers.Add(workPhone);
                        }

                        if (contactsCsv["Home Phone"] != "")
                        {
                            PhoneNumber homePhone = new PhoneNumber(contactsCsv["Home Phone"]);
                            homePhone.Rel = ContactsRelationships.IsHome;
                            contact.Phonenumbers.Add(homePhone);
                        }

                        if (contactsCsv["Mobile Phone"] != "")
                        {
                            PhoneNumber mobilePhone = new PhoneNumber(contactsCsv["Mobile Phone"]);
                            mobilePhone.Primary = true;
                            mobilePhone.Rel = ContactsRelationships.IsMobile;
                            contact.Phonenumbers.Add(mobilePhone);
                        }

                        if (contactsCsv["Work Fax"] != "")
                        {
                            PhoneNumber faxPhone = new PhoneNumber(contactsCsv["Work Fax"]);
                            faxPhone.Rel = ContactsRelationships.IsFax;
                            contact.Phonenumbers.Add(faxPhone);
                        }

                        if (contactsCsv["Home Address"] != "")
                        {
                            PostalAddress homeAddress = new PostalAddress();
                            homeAddress.Value = FormatAddress(false, contactsCsv["Home Address"], contactsCsv["Home Address 2"], contactsCsv["Home City"], contactsCsv["Home State"], contactsCsv["Home ZipCode"], contactsCsv["Home Country"]);
                            homeAddress.Rel = ContactsRelationships.IsHome;
                            contact.PostalAddresses.Add(homeAddress);
                        }

                        if (contactsCsv["Work Address"] != "")
                        {
                            PostalAddress workAddress = new PostalAddress();
                            workAddress.Value = FormatAddress(false, contactsCsv["Work Address"], contactsCsv["Work Address 2"], contactsCsv["Work City"], contactsCsv["Work State"], contactsCsv["Work ZipCode"], contactsCsv["Work Country"]);
                            workAddress.Rel = ContactsRelationships.IsWork;
                            contact.PostalAddresses.Add(workAddress);
                        }

                        if (contactsCsv["Organization"] != "")
                        {
                            Organization organization = new Organization();
                            organization.Label = contactsCsv["Organization"];
                            organization.Title = contactsCsv["Job Title"];
                            contact.Organizations.Add(organization);
                        }

                        StringBuilder notes = new StringBuilder();

                        if (contactsCsv["Web Page 1"] != "")
                            notes.AppendFormat("Website: {0}", contactsCsv["Web Page 1"]);

                        notes.AppendFormat("{0}{1}", notes.Length > 0 ? "\r\n" : "", contactsCsv["Notes"]);

                        contact.Content = notes.ToString();

                        Contact createdContact = cr.Insert(feedUri, contact);
                    }
                }
            }
        }

        static public string FormatAddress(bool useBR, string address1, string address2, string city, string state, string zip, string country)
        {
            StringBuilder address = new StringBuilder();
            string newLine = useBR ? "<br />" : "\r\n";

            if(address1 != "") address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   address1);
            if(address2 != "") address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   address2);

            StringBuilder cityStateZip = new StringBuilder();

            cityStateZip.Append(city);
            if(state != "")    cityStateZip.AppendFormat("{0}{1}", cityStateZip.Length > 0 ? ", "  : "", state);
            if(zip != "")      cityStateZip.AppendFormat("{0}{1}", cityStateZip.Length > 0 ? "  " : "", zip);

            if(cityStateZip.Length > 0) address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   cityStateZip.ToString());
            if(country != "")           address.AppendFormat("{0}{1}", address.Length > 0 ? newLine : "",   country);

            return address.ToString();
        }

    }
}

Question: Data Feed says 8660 of 8660 items inserted, but no Items (Active, Published or Inactive)

pjtpj Level 1 7/27/09

We have written a program that generates and uploads a tab delimited data feed file. This program was working fine for several months until around June 16. Starting around June 17, the program and the FTP upload has been working, but GoogleBase is not updating our GoogleBase items from the data feed file. We didn’t make any changes on our end and all required fields are in our data feed file. All our GoogleBase items eventually expired and were deleted last week.

Last Friday, I tried logging in to GoogleBase with my browser and manually uploading the data feed file. The data feed file was processed successfully, with a handful of the usual warnings and errors we always get (missing upc, encoding problems, missing discription). The missing UPC was a problem for all our products, but we only have 22 products with other errors. As before, I see the message “8622 of 8622 items inserted”, but I see no Items. Next, on Saturday, I tried adding a new Data Feed file. This time, the file got stuck “Processing” until today. Today, I finally got the error “Error: Internal server error while retrieving 9 # 0″. OK, so that is just some weird error.

Today, I changed our program to add UPCs where we have them and manually uploaded the data feed file again. I quickly got “8660 of 8660 items inserted – Processed on Jul 27, 2009 6:09 pm CST”, and the usual handful of warnings and errors (now I get some invalid UPC errors), but still I have no GoogleBase items.

I can manually add new Items and they become “Published and searchable”.

The Data Quality tab is basiclly empty (has information about the 1 item we created manually).

I tried creating a new GoogleBase account from scratch, but Google won’t let me create another account for the same web site (fair enough, I’m just trying to troubleshoot).

Any suggestions?

All answers

pjtpj Level 1 7/27/09

I just received a large batch of data feed related emails. The one associated with the most recent data feed file upload says:

No items inserted
Uploaded at: July 27, 2009 6:03:44 PM CST
Processed at: July 27, 2009 6:09:30 PM CST

For more details on the status of your data feed, go to http://www.google.com/base/showfeedsummary?fid=1336756

When I click on the link, I get the same “8660 of 8660 items inserted – Processed on Jul 27, 2009 6:09 pm CST” page I’ve already seen.

Celebird Top Contributor 7/27/09

what’s your website-url, google-base display-name and account-id?
has there ever been any disapproved messages seen within any account?

pjtpj Level 1 7/27/09

Website URL: http://www.mountainhighoutfitters.com
Display name: Mountain High Outfitters
Account ID: 2476386

No, we haven’t ever had any problems with disapproved items. There is one inactive/disapproved item right now – it’s the item with the title “Test Item 1″ I created on Friday to see if I could manually add an item.

Celebird Top Contributor 7/28/09

first, check the ‘view errors and suggestions’ page.

if there are no messages on the view-errors page –
you would need to contact google-base directly here:
http://base.google.com/support/bin/request.py?contact_type=error&ctx=direct

pjtpj Level 1 7/28/09

There is nothing the “view errors and suggestions” that would suggest why we have no items. As I mentioned earlier, we do have problems on 22 of the 8660 products, but there is nothing I would consider critical and nothing we weren’t seeing a month ago.

Today, I fixed a problem with custom attribute names (we were including spaces). This problem was always there, but I figured fixing it was worth a try. My upload just finished processing and it hasn’t seemed to help… still no items.

As you suggested, I filled out a support request. Thanks.

Kyle8 Level 1 7/28/09

Pjtpj / Celebird, I am experiencing exactly the same issue.  I upload our files and immediately get an email saying ‘No Items Inserted’ but when I click into my dashboard it says “x of x” (etc.) inserted successfully for all data feeds.  Under errors/suggestions it says I am missing the weight attribute but that the weight attribute is not required and will affect the feed. 

I have been experiencing this issue for over 2 weeks now.  I have been in contact with someone named Atanu at Google, & have sent well over 7 replies to him.  He has been somewhat responsive but each time just responds with a canned response saying “our engineers are working on resolving this issue. We sincerely apologize for the inconvenience.”  As I’m sure is the case with you, this has had a significant impact on our traffic and sales.  Has anyone else who is experiencing this problem found a workaround or a more solid response from Google?

eimaj Level 2 7/28/09

We had the same issue … identical, but our feed is Atom 1.0

We are making progress having successfully processed only 100 out of 30000+ products.  Little by little, we are adding more the the new feed and going from there.

Here’s what we did (for Atom 1.0 feed):
Having nothing to lose (our products expired as well), we deleted the feed file from Google Base dashboard
Made sure no products showed in www.google.com/base by authorid and by site:site.com search (count = 0)
Added a new feed file in the Google Base dashboard having a new unique file name different than all others used before
Created a new feed file having the exact content EXCEPT the //id was new, unique and different than all others used before – only having 2 products
Made sure those 2 products showed in www.google.com/base by authorid and by site:site.com search (count = 2)
Attempt at all 30000+ in a file failed … looked like we were back to square one
Attempt at 3000+ file failed … looked like we were still at square one
Attempt at 100 file succeeded!  We excluded products we did not have UPC or placeholder images, CDATA’ed the //entry/product_type and shortened the //entry/title in this file as well
We are at the point of progressively adding more products to the file trying to pinpoint the issue.

If anything is found, I will be more than happy to share the information. 

~Jamie

cheapercigars Level 1 7/28/09

ditto to pjtpj. I am facing the same issues with our feed just like pjtpj.
I have sent couple of emails to google.
still waiting for an answer.
lets hope its nothing serious.
- Mel

pjtpj Level 1 7/28/09

Thanks for all the replies. I haven’t heard anything back from Google, but I will post here when/if I figure anything out.

From the comments above, I suspect that the problem has something to do with the actual data in my data feed file. eimaj, maybe you can pinpoint the offending data via binary search: Divide the data in halves, then try to process each half separately. If one of the halves works, then divide the half that doesn’t work in halves (quarters of the original file) and try to process each new half separately and so on. This should at least be faster than trying 100 products at a time. Maybe something we added or updated around June 17 is causing the Google data feed file processing code to crash and/or reject all our items. Or maybe Google made some change to the code they use to process data feed files.

Here is the email I just got from today’s upload:

No items inserted
Uploaded at: July 28, 2009 7:37:05 AM CST
Processed at: July 28, 2009 8:10:40 AM CST

But, still, when I login to look at the data feed, I see “8660 of 8660 items inserted – Processed on Jul 28, 2009 8:10 am CST”.

This mismatch between the count of items inserted in the email and the web page is some kind of bug, but it doesn’t help point me to what is causing the problem. Just a little feedback from Google might go a long way…

Since others are reading this thread and having similar problems, I’m going to share some of my code and see if anybody else sees anything obviously wrong. Going with the theory that there is a problem in the data file, maybe there is a problem with the way I am encoding the file’s data fields. Here is the function I use to encode most data fields:

static char[] _quotedCharacters = new char[] { ‘\t’, ‘\r’, ‘\n’, ‘”‘ };

// GoogleBase specific rules
// See http://base.google.com/support/bin/answer.py?answer=25200&hl=en_US
public string TsvQuote(string s)
{
if (s.IndexOfAny(_quotedCharacters) >= 0)
s = “\”” + s.Replace(“\””, “\”\””) + “\””;

return s;
}

Here is the header line from my data feed file:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string

We use our own categories for product_type. The way the values are input into the system, the users can’t enter in non-printable characters, so I just do a simple search and replace for double quotes:

StringBuilder categoryText = new StringBuilder(category.Name);

while (category.ParentCategoryID != null)
{
category = Category.FindCachedCategory(category.ParentCategoryID.Value);
if (category == null)
{
categoryValid = false;
return “”;
}

categoryText.Insert(0, category.Name.Replace(“\””, “”) + ” > “);
}

return categoryText.ToString();

All the above code seemed reasonable at the time and was working OK until about June 16. OpenOffice has no problem parsing my file properly. I do have some HTML embedded into descriptions, and I’m not sure how GoogleBase treats that, but the HTML is not new.

pjtpj Level 1 7/28/09

Looking at my data feed file, I have a couple of ideas/thoughts:

1) I am worried about product_type. The GoogleBase specifications aren’t crystal clear… I coded up my best guess and it seemed to work, but maybe it really doesn’t. Here are some examples from my data feed file:

“Clothing > Men’s Clothing > Men’s Lifestyle Clothing > Men’s Lifestyle Pants”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $50 – $74.99″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $75 – $100″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”
“Gear > Camping/Hiking > Backpacks > Fast and Light Packs”,”Gear > Camping/Hiking > Backpacks > Weekend Packs”
“Accessories > Book Bags”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Activities > Adventure Travel > Travel Gear > Travel Packs”,”Gear > Camping/Hiking > Backpacks > Day Packs”
“Gear > Camping/Hiking > Backpacks > Day Packs”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Footwear > Men’s Footwear > Men’s Multisport Footwear”,”Activities > Climb > Climbing Shoes”,”Earth friendly Gear”
“Clearance > Winter Sale”,”Clothing > Kid’s Clothing > Kid’s Shirts”

As you can see, many of our products are in multiple categories / product_types and I believe we handle this correctly according to GoogleBase, but maybe somebody can see something I’m doing wrong.

2) I upload products with 0 quantity in stock. I know that these items are automatically deactivated, but that is what I want and it seems to be the most straightfoward way to make sure that products that go out of stock in our stores are removed/deactivated from GoogleBase.

cheapercigars Level 1 7/28/09

hello,
today in the morning I read that it may be a possibility that the file size is too big.
I read it in google help pages.
So, I divided my feed into two parts.
But no luck, the same thing happens. You can see the products being inserted but the email mentions that No Products Inserted. Also, when I checked the “data quality” tab, it does not show the new products.
So, if any of you get around it, please let me know on the thread. I am watching it.
Thanks
Mel

pjtpj Level 1 7/28/09

OK, so I think I am starting to narrow down the problem. Based on the idea from eimaj, I only kept the first 291 items in my data feed and eventually got back this email:

278 of 291 items inserted
Uploaded at: July 28, 2009 11:33:31 AM CST
Processed at: July 28, 2009 11:46:58 AM CST

And now a couple of hours later, I finally have 273 items “Published… searchable soon”. Between character encoding errors in description or quantity = 0, the missing items can all be accounted for. So, the problem seems to be somewhere in the content of my other 8369 items… Item # 292 has some Word HTML garbage – my current theory is that the source of the problem is with the HTML in some of the descriptions. I have figured out that some HTML is OK, because tables and bullet lists seem to make it through. One idea is that if any of the descriptions contain certain tags (<head>, <title>, <style>, <script>, etc.), then some final stage of data feed processing crashes and the entire feed ends up in the bit bucket.

cheapercigars Level 1 7/28/09

hello pjtpj
i understand that you had some “html code” in one of the line.
but my feed is strictly text, tab deliminated.
i did check it before posting this reply.
any more inputs will be appreciated.
I am just hoping that google might have fixed this issue (if there was something wrong with them to begin with)
Thanks
Mel

username6 Level 1 7/28/09

Did you try to submit data feed without custom attributes? We have exactly same problem. All items items just disappeared from active items tab.

cheapercigars Level 1 7/28/09

yes we did
but we really get high rankings when we have custom attributes.
I just heard from couple of good sources that FTP does not work.
This time I have tried to schedule the feed and let google fetch by itself.
lets see how this works.
i’ll keep it posted
Thanks
Mel

pjtpj Level 1 7/29/09

The 291 items that successfully processed yesterday had custom attributes. You can see the custom attribute I’m using in the header from my data feed file I posted above.

Kyle8 Level 1 7/29/09

Cheapercigars, if it’s the FTP that you’ve been hearing is down, do you know if the API is working? Anyone using the API just fine?

pjtpj Level 1 7/29/09

This morning, I created and uploaded a data feed file that avoided any product that had <head <title <style <script or <meta in its description. That did not work. Now I have taken that data feed file and broken it into 4 parts. I am processing all 4 parts right now. Maybe 1 or more part will succeed and help me pinpoint the source of the problem.

pjtpj Level 1 7/30/09

Yesterday, I worked it down to a data feed file with only 9 items that causes the problem. I have tried 3 different possible guesses as to what in the file is causing the problem, but I haven’t figured it out yet.

This morning, I got a fairly generic email response from GoogleBase Support:

Thank you for your email. I have reviewed your Google Base account and the
reason few items your file are failing is because your feed contains
unsupported characters. For example, certain white space characters, such
as tabs, may not process correctly and will result in a feed failure.

To correct this error, please remove all unsupported characters like HTML
tags etc from your feed and upload your file again.

Please let me know if you have any further questions or concerns.

Unfortunately, this response was not helpful. I already knew about the “few items” and, as mentioned here, I have submitted multiple data feed files that don’t have any of these issues, but still fail to update my GoogleBase items.

pjtpj Level 1 7/30/09

I have finally narrowed this problem down to just 1 product/item. Here is a copy of the data feed file that will not process successfully:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
“<ul>
<li>A multipurpose rescue/rappel rope.</li>
<li>100% nylon low stretch static.</li>
<li>Uniform sheath coverage for durability and gear compatibility .</li>
<li>Ideal for heavy duty alpine, big wall, and expedition fixing and haul lines.</li>
<li>Great for mountain rescue and industrial rope access.</li>
<li>Certified to NFPA 1983, 2006 edition for Life Safety Rope forlight use.</li>
</ul>” 9995 http://www.mountainhighoutfitters.com/Products/SuperStatic+7%252f16%2522+-+300ft.html 255.00 “Sterling Rope SuperStatic 7/16″” – 300ft” Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-1.teztech.com/Products/7662.jpg 8.00 pounds 1 new

My current theory is that it is the URL that is tripping it up. This URL is valid (works in IE, FF and Chrome), but it does contain URL encoded characters ( / and ” ) that could cause problems if unescaped. The other interesting thing about this product is that it was added to our database 14 May 2007, so it was working OK at some point (we were able to upload all our products to GoogleBase successfully in the past).

Celebird Top Contributor 7/30/09

i would not include any html.

pjtpj Level 1 7/31/09

I made a mistake in my last post. Actually, that data feed file in that post, even with the HTML and complex URL will process OK. Here is the file that will not process successfully:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
The Nose in a day, dogging your project for the painful 20th day, and everything in between, this 10.4mm rope handles all ends of the spectrum. Lightweight, it will never slow you down for a second and will always return day after day to handle the abuse your projects dish out. 39705 http://www.mountainhighoutfitters.com/Products/Marathon+Sport+70m.html 224.00 Sterling Rope Marathon Sport 70m – 70m Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-2.teztech.com/Products/7661.jpg 7.00 pounds 2 new 70m

There are no double quotes around the description because it shouldn’t need them.

I am going to submit this file again right now just to be sure.

PJ

Celebird Top Contributor 7/31/09

not including html was more a policy recommendation not about formatting –
but html can cause formatting issues as well so isn’t recommend at all.

quotes cannot be determined to be “needed” or “not needed”
without knowing what the quoted-fields setting is for the feed –
each feed can be set differently and effect quotation requirements.

without a url link to the file and knowing the settings for the registered-feed
the copy-and-paste does not reflect what google-base is doing with the file –
the same is true for issues such as encoding.

also, currently, only one product_type value is supported in tab-delimited feeds.

pjtpj Level 1 7/31/09

I haven’t seen anything that says if HTML is allowed or not. I have noticed that HTML seems to be stripped from my data feed, but is allowed for manually submitted items. I have not noticed that HTML prevents items from getting posted (or, as in my case, causes some internal problem with data feed file processing).

Tab and comma delimited files existed long before Google decided to allow their use for data feed files. My understanding has always been that, in these types of files, a field “needs” to have quotes around it if it contains the delimiter character or line feed. Here is what Google has to say about this subject: http://base.google.com/support/bin/answer.py?hl=en_US&answer=25200. It is not explained how the use of quotes interacts with the  “Use Quoted Fields” setting, but I always set this to “Yes” under the assumption that “Yes” allows the use of quotes around fields (but does not require it). I have many successful runs that use the code I posted above, so I think, at least to some degree, GoogleBase also understands that fields within a tab delimited file may or may not have quotes around them, depending on if they are “needed” or not.

I am still working on this (and just hoping that maybe somebody else sees something obvious)… if I find out for sure that the data feed file I’m working on now with just a single product is causing the problem, then I will post the file for downloading. Here are the settings I use for all my data feeds:
Feed format: googlebase, File format: text, Encoding: autodetect (detected as UTF-8), Delimiter: tab, Use Quoted Fields: Yes

Where did you read that only one product_type value is supported in tab-delimited feeds? This page seems to suggest that you can have more than one: http://base.google.com/support/bin/answer.py?hl=en&answer=66818. Looking at the raw data feed file and some items that have gotten through, I figured out that I probably need to quote the comma separated product_type values individually rather than trying to quote the entire field (right now, all my product_type values end up with double-quotes around them). But this is a minor problem and I am failure sure it’s not the cause of whatever is keeping my big file from getting processed.

Celebird Top Contributor 7/31/09

i wouldn’t include any html in feeds.

google-base feed quotes exist within the context of
feed settings apart from any tab-delimited standards.

i believe use-quoted-fields set to “no” has always meant
to turn-off the feature so quotations will be seen as part
of the value — set to “yes” the feature is operational so
quotations will be used as a field delimiter, not a value.

also, length is a google-base defined attribute so
probably shouldn’t be redefined as a custom-attribute.

what exactly does the ‘view errors and suggestions’ page display?

understood about the help information and the product_type attribute — see also:
http://www.google.com/support/forum/p/base/thread?tid=476189d91e13bdaa&hl=en
http://www.google.com/support/forum/p/base/thread?tid=77a9e517542cee97&hl=en

pjtpj Level 1 7/31/09

I’ve uploaded the same file (with only one product) twice in a row now with the same problem occurring, so I think I have it down to a repeatable test case. Here is a link to the file: http://teztech.com/files/GoogleBaseProducts4F1.txt. I uploaded the file in binary mode to make sure it was the same as what I am uploading to Google. After I upload the file and it is processed, here is what GoogleBase says when I login:

1 of 1 items inserted – Processed on Jul 31, 2009 7:20 am CST
Detected encoding: US-ASCII
Detected delimiter: Tab (\t)
Detected file format: Text
Feed Level Messages
Warning: Missing recommended attribute: upc
While items missing recommended attributes will process successfully, we recommend including relevant attributes if they are available.

And here is what the email from GoogleBase says:

No items inserted
Uploaded at: July 31, 2009 6:33:37 AM CST
Processed at: July 31, 2009 7:20:04 AM CST

The “No items inserted” email is how I determine that “the problem” has occurred. I have also noticed that it takes a relatively long time to process these files. I’m sure they go into a queue and the processing time depends on how busy the GoogleBase data feed servers are, etc., but I’ve been processing these files manually for 5 days in a row and I think an hour to process a 1 product data feed file is longer than usual.

The GoogleBase data feed documentation explains how to create tab delimited files by hand in Excel and OpenOffice Calc, so I would assume that, at least to some degree, the GoogleBase programmers had the output of these programs in mind when they created their import code.

Celebird, I agree that “length” probably shouldn’t be a custom attribute. Thanks for the second pair of eyes. I’m going to try making some changes to length and see what happens. Interestingly, the length in my data feed is the length of the product (a rope in this case) and not the length of the shipped package. I wonder which one Google means by “length”? If they can use length to calculate shipping charges, then it is the length of the shipped package. I’ll try and research this.

Thanks for the pointers to the product_type threads. I just received an email with “Our engineers are aware of this problem and are working to resolve it.” on my issue, so, unfortunately, this seems to be some kind of standard response.

pjtpj Level 1 7/31/09

1 person says this answers the question:

Changing the name of my length field from c:length:string to c:product_length:string allowed the file with 1 product to be successfully processed. Now let’s see if I can process a real data feed file.

ajj Level 1 7/31/09

Hi, in case the products are still not showing try this:  I see you’re using multiple categories – hence the reason for my recommendation:

1. On the google base dashboard, click My items tab.  Then click Data feeds.  Next click the edit link adjacent to you data feed file so you can set or verify the base configuration.

2. You should be in the “Edit Data Feed” screen at this stage.  Verify the following (or adjust to have them)

File format:                     Text
Encoding:                       Autodetect
Delimiter:                        Tab
Use Quoted Fields:          No

Note, the last item should a No and NOT Yes in your case.  Then resubmit.  Hope this helps

pjtpj Level 1 10:51 AM

Here is the source of my problem: GoogleBase processing crashes when I have a custom attribute named “c:length:string” and a value of “70m” (I assume other non-empty values cause problems as well). I changed my custom attribute name to “c:product_length:string” and now, the email I just got back from Google says: “8584 of 8606 items inserted”.

Some might say that this was an obvious problem, but our Internet store allows our end users to create their own attribute groups (color, size, style, etc.). When we create a data feed file, there is some code that matches our attribute groups to standard Google attributes (color, size). If there is not a match, then we automatically create a custom attribute for GoogleBase. In our case, “length” in our data base is the product length, not the shipping length, so using a custom attribute actually make sense.

In addition, I ran into other issues that caused a simple problem like this to take an entire week to work out:

- Even though GoogleBase processing was failing, the data feed message on the web site said “8660 of 8660 items inserted”. The GoogleBase data feed processing code was crashing on my data feed file, but apparently GoogleBase doesn’t code in place to trap these types of errors and provide useful diagnostics to outsiders like me.
- In my 8660 product data feed file, only a few products caused the problem and there was no easy way to narrow down the source of the problem. I had to manually divide my file into parts, submit the parts manually, then wait on the “success” emails to arrive, then rinse and repeat.
- There was nobody at GoogleBase support with the willingness or ability (not sure which) to help me troubleshoot this issue.
- Documentation for data feeds is just vague and wrong enough to keep you guessing. The biggest documentation issue I see right now is with product_type and how it interacts with the Use Quoted Fields setting.

If you run into a similar problem, the first step is to make sure you are getting all the data feed emails. Only the email said “No items inserted”. Next, you must isolate the data that is causing the problem. I assume my problem with a custom attribute named “c:length:string” is pretty specific, but the technique I ended up using to isolate it is general (if slow and painful).

Celebird Top Contributor 11:33 AM

these are many of the same similar reasons why it seemed a good idea
to write a feed validation checker with an ever expanding set of checks –
http://celebird.googlepages.com/ckbase.htm

also, 70m is proabably an invalid standard-length value — a valid length would be 70 m

“standard” attributes can be google-defined attributes but
are in the context of an item-type; not all defined-attributes
are allowed or are valid within the context of an item-type –
an obvious example is location; item-type requirements are
well-defined and i believe must be taken into account if any
private-code is creating the final set of attributes and values.

google-base processing has always been more simple counts
and basic formatting rather than robust validation; this is most
evident with respect to encoding, quoted-attributes and other
per-feed features — i believe this is one reason why the feed’s
suggestions feature was introduced.

pjtpj Level 1 12:10 PM

GoogleBase actually does an OK job of identifying some problems (with the suggestions feature). If GoogleBase would catch and report errors like mine that cause a complete crash (I assume) and include the data feed file line number at the time of the crash, that would be a big help.

Celebird, I appreciate the effort on both the checker and on this thread -it was your comment that led me to the solution. I didn’t know about your checker. I just now submitted my test file (http://teztech.com/files/GoogleBaseProducts4F1.txt) and it passed with warnings. You could add a test to check for custom attributes with the names of standard attributes. That test file also has lots of quotes within the product_type field… from my own tests and comments from others, it looks like if you want to use multiple categories, then you have to set “Use Quoted Fields” to “No” (which then won’t allow newline characters in the description because you can’t use quotes to delimit a field). You could add a check for this as well, but actually, I hope GoogleBase cleans up their act on defining and parsing this field. XML is such a waste, I really don’t want to switch file formats just for this, but I also admit that text files have their own problems.

Yes, it’s possible that “70 m” vs. “70m” could have fixed the problem as well. On my side, “length” is a attribute group defined by my customers and the values entered are free text strings (which makes sense for rock climbing ropes that come in 50m, 60m and 70m), so the safest course for me was to rename the field. At some point, I plan to add support for dimensional shipping. When I do that,I’ll go back and figure out what Google means by “length” and rework things accordingly.

Recently, I upgraded my development system from OpenOffice 2.x to OpenOffice 3.1. I can’t remember now the main reason I upgraded, but I was looking forward to being able to add the ability to convert docx to pdf (Office Open XML support was added in OpenOffice 3.x). I figured the upgrade might require some minor changes to my document conversion code, but it turned out not to be so simple.

The first problem was that OpenOffice no longer showed up under “Add Reference” in Visual Studio. After some Google searches, I finally stumbled across a solution. OK, wow, digging though install cab files for DLLs, hacking the registry and manipulating the PATH is ugly, but it worked – at least once. Almost as soon as my test program ran successfully, Visual Studio locked up. The only way to get control back was to kill the Visual Studio process, then restart it. Every time I opened the solution file, after just a few seconds, Visual Studio would lock up again. Removing every trace of the OpenOffice DLL references would prevent the lockups, but without OpenOffice, I could not convert documents. I eventually figured out that the problem was that the library project that used OpenOffice was also included in my web site project. Because the web site uses the library, too, the OpenOffice reference is automatically copied to the web site project. Normally, this would be no big deal – just a couple of small DLL files in the web site bin folder that would not be used – but in this case, Visual Studio could not successfully build the web site project as long as the OpenOffice DLLs were referenced.

Now that using OpenOffice was no longer as simple as “Add Reference”, it was time to take a look at a way to decouple my application from the OpenOffice DLLs. The API my application needs is pretty simple: Given the bytes of a file in Doc (or .docx) format, return the bytes of the file in Pdf format. Also, there are security risks whenever you open a file in a complicated piece of software like OpenOffice. These two facts pointed me to turning the Pdf conversion into a web service. The client application and web service can be partitioned across computers, so, if I wanted to, I could run the Pdf conversion service on a virtual machine with its own install of OpenOffice and tight security and resource control. The web service API didn’t need fancy XML: Clients could post the Doc file via a standard HTTP MIME encoded form. The result would be a the PDF byte stream returned via HTTP. This would make the service easy to develop, test and use from clients on many platforms.

I typically develop my large web based applications in C# and Asp.Net. However, for small services like document conversion, I often use PHP. Simple PHP applications can be just a file or two and I can move a PHP based service over to any web server that supports PHP. So, my first attempt at a document conversion service was written in PHP and used the OpenOffice COM API. Using my earlier OpenOffice API code as a guide and some PHP sample code I found on the web, I was able to quickly create a command line PHP application that could perform Pdf conversions. With this proof of concept working, I coded up the PHP web service. But the web service wouldn’t work. I could see in Task Manager that the OpenOffice executables were launched, but any code running under the web server that tried to access the OpenOffice COM objects would freeze up.

I spent a lot of time trying different security settings for various files and DCOM. I even tried creating a dedicated user and configuring DCOM to run OpenOffice as the dedicated user when the OpenOffice COM objects were activated. I changed Windows event log settings to log all security violations. Nothing would work and I couldn’t see anything useful in Event Viewer. I tried coding up a simple Asp.Net application and got the same results under the .Net OpenOffice API. Under both PHP/COM and C#/.Net, OpenOffice could be automated by a command line program but would not run under IIS. Right now, I develop on Vista and deploy to Windows Server 2008, so one might think the problem was trying to use IIS 7, but OpenOffice 2 didn’t have similar problems.

Finally, I gave up trying to run OpenOffice directly under IIS. The problem was either some type of security issue or that OpenOffice was trying to pop up registration dialogs. In Windows, the simple way to run a program under a given user account is to run the program as a Windows Service. It turns out that it is pretty easy to create a Windows Service in C#. The basic idea I came up is was:

1. Install OpenOffice
2. Create a Windows user account dedicated to Pdf conversion.
3. Login as the Pdf conversion user and make sure that OpenOffice opens without prompting for registration, crash feedback, etc. (I suspect that the reason OpenOffice 3 won’t run properly under IIS is that it is prompting the user to register).
4. Create a Windows Service with a simple API to convert documents
5. Install the Windows Service and configure it to run as Pdf conversion user
6. Access the Windows Service from my document conversion Web Service

Here is how the API calls flow: Application -> Web Service -> Windows Service

As I mentioned earlier, the API between the application and the web service was standard a HTTP Post. Between the Web Service and the Windows Service, I originally planned to use a simple, proprietary TCP/IP protocol. As long as you aren’t try to get too fancy coding thread pools and asynchronous I/O, at least in C/C++, it’s pretty easy to open a TCP socket and communicate between client and server with a line oriented protocol (like HTTP, POP3, SMTP, etc.). But, since both the Web Service and the Windows Service were going to run on Windows (at least for now), I decided to take a look at .Net Remoting. The .Net remoting solution was easy to code and use so I ended up sticking with it.

Below is some code to show you how everything fits together:

The C# interface IConverter is compiled into a DLL. The DLL is referenced by the client (Web Service) and server (Windows Service).

    public class ConverterEndpoints
    {
        public static int    Port = 7047;
        public static string Uri  = "DocumentToPdf";
    }

    public interface IConverter
    {
        void Convert(string srcDocument, string destPdf);
    }

The class ConverterEndpoints holds a couple of constants that the client and server use to setup the TCP port and service name. The code below demonstrates how ConverterEndpoints is used in the Windows Service. I’ve included code for the entire class so you can see how easy it is to program a Windows Service and access the Windows Event Log in C# – no 3rd party libraries are required.

The original version of this interface passed the files as simple byte[] variables. Theoretically, the files never need to be written to disk (input and output is via HTTP). But, I found that I could easily pass small byte[] variables across .Net remoting but large byte[] variables failed. Since the Web Service and the Windows Service run on the same machine and the file has to be written to disk for OpenOffice to do the conversion, I settled for creating temporary files and passing the names of the files between the Web Service and Windows Service.

   class WindowsService : ServiceBase
    {
        protected EventLog _log;
        protected TcpChannel _channel;

        public WindowsService()
        {
            ServiceName = "Teztech Document Conversion";
            EventLog.Source = "Teztech Document Conversion";
            EventLog.Log = "Application";
            
            // These Flags set whether or not to handle that specific
            //  type of event. Set to true if you need it, false otherwise.
            CanHandlePowerEvent = false;
            CanHandleSessionChangeEvent = false;
            CanPauseAndContinue = true;
            CanShutdown = true;
            CanStop = true;

            if (!EventLog.SourceExists("Teztech Document Conversion"))
                EventLog.CreateEventSource("Teztech Document Conversion", "Application");

            _log = new EventLog();
            _log.Source = "Teztech Document Conversion";

            _channel = new TcpChannel(ConvertInterface.ConverterEndpoints.Port);
            ChannelServices.RegisterChannel(_channel, false);

            RemotingConfiguration.RegisterWellKnownServiceType(typeof(DocumentToPdf), ConvertInterface.ConverterEndpoints.Uri, WellKnownObjectMode.Singleton);
        }

        static void Main()
        {
            ServiceBase.Run(new WindowsService());
        }

        /// <summary>
        /// Dispose of objects that need it here.
        /// </summary>
        /// <param name="disposing">Whether or not disposing is going on.</param>
        protected override void Dispose(bool disposing)
        {
            base.Dispose(disposing);
        }

        protected override void OnStart(string[] args)
        {
            _log.WriteEntry("Service Starting");

            _channel.StartListening(null);

            base.OnStart(args);

            _log.WriteEntry("Service Running");
        }

        protected override void OnStop()
        {
            _log.WriteEntry("Service Stopping");

            _channel.StopListening(null);

            base.OnStop();
        }

        protected override void OnPause()
        {
            _channel.StopListening(null);
            base.OnPause();
        }

        protected override void OnContinue()
        {
            _channel.StartListening(null);
            base.OnContinue();
        }
    }

Below is the code behind file for the Web Service client. As you can see, .Net makes this type of Web Service easy to create.

public partial class Pdf : System.Web.UI.Page
{
    protected TcpChannel _channel;

    protected void Page_Load(object sender, EventArgs e)
    {
        try
        {
            if (Request["Username"] == null || Request["Password"] == null)
                throw new ApplicationException("Username and Password are required");

            HttpPostedFile postedFile = Request.Files["InputFile"];
            if (postedFile == null || postedFile.FileName == null || postedFile.FileName == "")
                throw new ApplicationException("InputFile was not supplied or file name is empty.");

            string inputFileExtension = Path.GetExtension(postedFile.FileName);
            if (inputFileExtension.Length < 2)
                throw new ApplicationException("InputFile does not have a valid file name extension.");
            inputFileExtension = inputFileExtension.Substring(1);

            string outputFileName = Request["OutputFileName"];
            if (outputFileName == null || outputFileName == "")
                outputFileName = Path.GetFileNameWithoutExtension(postedFile.FileName) + ".pdf";

            DbRequest.CheckAuthorization(Request["Username"], Request["Password"]);

            if (_channel == null)
            {
                Dictionary<string, string> channelProperties = new Dictionary<string,string>();
                channelProperties["name"] = "";
                _channel = new TcpChannel(channelProperties, null, null);
                ChannelServices.RegisterChannel(_channel, false);
            }

            string url = string.Format("tcp://localhost:{0}/{1}", ConvertInterface.ConverterEndpoints.Port, ConvertInterface.ConverterEndpoints.Uri);

            ConvertInterface.IConverter converter = (ConvertInterface.IConverter)Activator.GetObject(typeof(ConvertInterface.IConverter), url);

            string path = MapPath(".");
            path = Path.Combine(Path.GetDirectoryName(path), "Temp");

            using (TempFileCollection tempFiles = new TempFileCollection(path))
            {
                string inputFile  = tempFiles.AddExtension(inputFileExtension);
                string outputFile = tempFiles.AddExtension("pdf");

                postedFile.SaveAs(inputFile);
                converter.Convert(inputFile, outputFile);

                Response.Clear();
                Response.ContentType = "application/pdf";
                Response.AddHeader("content-disposition", string.Format("inline; filename={0}.pdf", "sample"));

                byte[] pdfBytes = File.ReadAllBytes(outputFile);

                Response.OutputStream.Write(pdfBytes, 0, pdfBytes.Length);
            }
        }        
        catch (Exception ex)
        {
            Response.Clear();
            Response.Write(string.Format("<HTML><HEAD></HEAD><BODY><H1>Conversion Error</H1><P>{0}</P></BODY></HTML>", ex.Message));
            Response.StatusCode = 500;
            Response.StatusDescription = ex.Message;
        }
    }

For completeness, I have included the code for the actual Web Service’s .aspx file below. As you can see, it’s just standard .aspx. boilerplate – all HTTP output is generated in the code behind file’s Page_Load event (which will be invoked when the Web Service client posts to the page). This is quick and dirty, but it works.

<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Pdf.aspx.cs" Inherits="Pdf" EnableViewState="false" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    
    </div>
    </form>
</body>
</html>

Here And here is the code for a little PHP application I used to test my conversion Web Service:

<?php $ErrorMessage = ''; ?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>
File Conversion Server Demo
</title>
</head>
<body>

<h1>File Conversion Server Demo</h1>

<form name="Demo" method="POST" enctype="multipart/form-data" action="/demo.pdf">

    <?php if($ErrorMessage): ?>
    <p><font size="5" color="#008000"><?php echo $ErrorMessage; ?></font></p>
    <?php endif; ?>
    
    <table>
        <tr>
            <td><b>Username:</b></td>
            <td><input name="Username" value=""></input></td>
        </tr>
        <tr>
            <td><b>Password:</b></td>
            <td><input name="Password" value=""></input></td>
        </tr>
        <tr>
            <td><b>Input File:</b></td>
            <td><input type="file" name="InputFile"></input> </td>
        </tr>
    </table>
    
    <input type="submit" value="Submit"></input> 
    
</html>

If you look the PHP form’s action tag, you’ll see I used /demo.pdf as the action. Using the very nice IIS7 URL Rewrite Module, I map all *.pdf URLs to my conversion Web Service. OK, this might be a little overkill, but this way the URLs look nice and all end in .pdf (which is appropriate).

The final piece of the puzzle is a small wrapper class I created to access the conversion Web Service from within my C# application:

    public class ConvertClient
    {
        public ConvertClient(string convertHost, string username, string password)
        {
            _ConvertHost = convertHost;
            _Username    = username;
            _Password    = password;
        }

        protected string _ConvertHost;
        public string ConvertHost { get { return _ConvertHost; } set { _ConvertHost = value; } }

        protected string _Username;
        public string Username { get { return _Username; } set { _Username = value; } }

        protected string _Password;
        public string Password { get { return _Password; } set { _Password = value; } }

        protected string _Response;
        public string Response { get { return _Response; } set { _Response = value; } }

        protected int _MaxConvertAttempts = 2;
        public int MaxConvertAttempts { get { return _MaxConvertAttempts; } set { _MaxConvertAttempts = value; } }

        protected string _ResponseCode;
        public string ResponseCode { get { return _ResponseCode; } set { _ResponseCode = value; } }

        protected static string _szBoundary    = "SEPARATORSTRINGTEZTECHDOTCOM1";
        protected static string _szBoundary2   = "\r\n--SEPARATORSTRINGTEZTECHDOTCOM1\r\n";
        protected static string _szBoundary3   = "\r\n--SEPARATORSTRINGTEZTECHDOTCOM1--";
        protected static string _szFileSizeHdr = "Content-Disposition: form-data; name=\"MAX_FILE_SIZE\"\r\n\r\n";
        protected static string _szFileHdrFmt  = "Content-Disposition: form-data; name=\"InputFile\"; filename=\"{0}\"\r\nContent-Type: application/octet-stream\r\n\r\n";

        public byte[] ConvertToPdf(byte[] inputFileBytes, string inputFileName, string outputFileName)
        {
            string url = string.Format("http://{0}/{1}?Username={2}&Password={3}", ConvertHost, outputFileName, Username, Password);

            // Calculate upload data size

            string szFileSizeData = string.Format("{0}", inputFileBytes.Length + 50000);
            string szFileHdr      = string.Format(_szFileHdrFmt, inputFileName);

            ASCIIEncoding ascii = new ASCIIEncoding(); // At this time, file names must be ascii
            List<byte> header = new List<byte>(); 
            List<byte> footer = new List<byte>(); 

            header.AddRange(ascii.GetBytes(_szBoundary2));      // MAX_FILE_SIZE field
            header.AddRange(ascii.GetBytes(_szFileSizeHdr));
            header.AddRange(ascii.GetBytes(szFileSizeData));
            header.AddRange(ascii.GetBytes(_szBoundary2));      // userfile field
            header.AddRange(ascii.GetBytes(szFileHdr));

            footer.AddRange(ascii.GetBytes(_szBoundary3));
            
            int cbContent = header.Count + inputFileBytes.Length + footer.Count;

            int attempts = 0;

            while (true)
            {
                try
                {
                    attempts++;

                    HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
                    webRequest.Method        = "POST";
                    webRequest.ContentType   = string.Format("multipart/form-data; boundary={0}\r\n", _szBoundary);
                    webRequest.ContentLength = cbContent;

                    using (Stream request = webRequest.GetRequestStream())
                    {
                        request.Write(header.ToArray(), 0, header.Count);
                        request.Write(inputFileBytes, 0, inputFileBytes.Length);
                        request.Write(footer.ToArray(), 0, footer.Count);
                    }

                    using (HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
                    {
                        using (BinaryReader reader = new BinaryReader(webResponse.GetResponseStream()))
                        {
                            byte[] pdfBytes = reader.ReadBytes((int)webResponse.ContentLength);
                            return pdfBytes;
                        }
                    }
                }
                catch
                {
                    if (attempts >= MaxConvertAttempts)
                        throw;
                }
            }
        }
    }

I didn’t see an easy way to build a MIME document in .Net, so I’m building up the MIME document (used in the HTTP Post) from scratch.

To debug the Windows Service, as shown in the code above, I added code to log to the Windows Event Viewer. In this case, I started with working document conversion code, so I didn’t end up needing to run the service under a debugger. But, one of the nice things about services running as a dedicated process (unlike Apache modules, ISAPI modules and Control Panel applets) is that, with just a simple change (adding a Main function), you can run the service as a regular command line application and attach the Visual Studio debugger to the running process.

To troubleshoot the Web Service, my first step was to disable “friendly errors”. With IIS7, you have to do it on both the web browser client and in IIS7. Debugging is just a matter of attaching the IIS7 process in Visual Studio (as usual for any web application).

All our production Windows servers use the 64 bit edition of Windows 2008 Server. Right now, I happen to do most of my development on a 32 bit edition of Windows Vista. Both platforms run IIS7 and usually, the 32 vs. 64 bit doesn’t cause any problems. However, the Windows version of OpenOffice only comes in a 32 bit version. This version runs fine on 64 bit Windows, but when programming OpenOffice via .Net, you need a 32 bit application, so you have to set the Target Platform in the Project Build Properties to X86.

For me, the IDE goes back to the days of Borland’s Turbo Pascal and Turbo C. They didn’t have GUIs much less GUI form editors, HTML wasn’t yet invented and I don’t remember the text editor being anything special. What turned them into IDEs was the super efficient work flow… A typical “Hello World” program was mostly a matter of typing in some code and pressing the Run key. If you wanted to debug, you would set your breakpoints right in the editor, then press the Debug key to start your program under the integrated debugger. When your program expanded to more than one file, IDEs had simple project facilities that did a good job of bridging the gap between “Hello World” and tools for large projects like “make” and “nmake”. Later, when the GUI APIs got so big that nobody could keep them all their head, integrated documentation became critical to a smooth workflow. For better or worse, I’ve become addicted to code completion. It doesn’t work as well for dynamic languages like PHP, but some code completion is better than none.

So, when I think IDE, I think of an all purpose application that makes a programmer’s day-to-day workflow smooth and efficient… I find that anything I can do that cuts down my “cycle” time improves my productivity (and my mood).

Every since the days of Microsoft C 6, Microsoft has always had the best IDEs for C/C++ and Windows development. Their documentation is better and better integrated, their compilers are way, way faster and the general work flow is always the smoothest. Visual Studio continues this trend for C/C++ and now C# development – it sure is easy to crank up a new C# web application and start debugging.

As as web host, I end up working on a lot of different sites. These days, there is a lot of PHP code out there and I end up spending a fair amount of time working on PHP projects. For most my really big projects, I typically use C# and Asp.Net, so I have Visual Studio handy. With a few registry tweaks to syntax highly PHP code, Visual Studio works OK as a text editor for PHP, but I wouldn’t call it a true PHP IDE. For one thing, there is no integrated debugging support. You can kind of get project file support (PHP files can be listed), but the only integrated documentation is for HTML and JavaScript (and sadly, the Microsoft documentation on these frequently referenced topics is IE centric).

So, several times, I’ve set out on a mission to find a IDE for PHP programming. Last time around, I gave up and stuck it out using Visual Studio as a text editor. After this last round of reviews, I ended up buying NuSphere’s PhpED.

When you are in the world of “free” software, it is tempting to stick only to “free” tools. However, in the case of IDEs, I just couldn’t find a free one that worked well enough to be better than Visual Studio.

One area where PHP needs a lot of work is support for debugging. The basics are there, but having to fiddle with various debugging DLLs is a hassle – debugging information has been standard for binary programs for years – how hard could it be to figure out a standard debugging API for IDEs to use?  Having to use query strings to initiate a debugging session is a serious design mistake. With all the rewriting and other things going on, fiddling with URLs just doesn’t make sense. Visual Studio is able to attach a debugger to server code in just about any situation – PHP and its IDEs should make this as easy.

The ability to automatically reformatting HTML in both blocks and full pages is very handy… plenty of tools (like content management systems) make a real mess of HTML. And, for better or worse, you end up using some of these tools over and over so it’s not like you can fix the code just once and be done with it.

Below are some notes I made as I tried out various PHP IDEs:

Eclipse PHP

No cost and open source

Cross-platform (Java)

Attractive

Amazingly slow and memory hungry – based on Java and requires a Java Runtime to be installed.

Integrated PHP reference uses the php.net web site so, it requires Internet access and is slow.

No integrated CSS, HTML, JavaScript or DOM reference

No WYSIWYG HTML editor, no integrated page preview, either

HTML reformat is weak and not customizable

Go to definition doesn’t work reliably

Code completion doesn’t work reliably

Code completion doesn’t work in CSS files.

HTML code completion uses upper case for tags and attributes.

Subversion integration is slow and shows incorrect status

Crashes frequently

Debugger worked for a simple page, but did not work in a real project

Komodo IDE

Attractive 

Cross-platform – interesting choice of Firefox engine

Reasonable performance and memory usage

Subversion integration is quick and accurate

Once configured, PHP syntax checking is very handy

No WYSIWYG HTML editor, but the integrated page preview is well done

Go to definition doesn’t work reliably

Code completion doesn’t work reliable (does not seem to follow include paths).

No integrated PHP, CSS, HTML, JavaScript, DOM or MySQL reference

Copy and paste of files in Projects tree fails with no error message

Deluxe syntax PHP + HTML syntax highlighting.

View unsaved changes off File menus is a great idea! The integration of this feature with the recovery of unsaved changes after a crash is nice, too.

Extremely annoying rendering bugs in the text editor

Code completion for CSS and HTML

CSS code completion is not very good. For example, once you type “margin:”, you get a drop list of choices instead of a description of the arguments for the margin style.

HTML code completion is annoying because it defaults to upper-case tags (is anyone still NOT using xhtml for new code?)  and does a poor job of suggesting closing tag (typing “</” does not give a drop list of closing tags).

PHPEdit

Attractive

Reasonable performance and memory usage

Subversion integration does not work out of the box.

Integrated PHP and MySQL reference is nice, no integrated CSS, HTML, JavaScript or DOM reference.

The ultra-deluxe syntax highlighting changes the highlighting rules depending on where the caret is placed – very handy for files with integrated HTML, PHP, CSS and JavaScript!.

No code reformat for any language other than PHP.

No code completion for any language other than PHP (CSS and HTML would be nice).

Go to definition (“Jump to declaration”) doesn’t work across files.

Cannot right click to open a file via a simple require() declaration.

No heroics to resolve PHP include paths for code completion.

DB Explorer is an interesting idea, but I think the most important thing about a database to a programmer is the column definitions (names, types, lengths) and, other than seeing the column names in the DB Explorer tree, there is no way see the column definitions for a table. Manually inputing a “show columns from” query into the SQL tab didn’t work – you end up with an “Invalid SELECT statement” error message in a message box.

Non-standard Alt+Tab behavior is not very useful.

PhpED

The debugger actually works! But, it’s a lot more hassle than it should be.

CSS code completion is OK, but Visual Studio is still the best.

PHP code completion is reasonable and follows include paths in some cases.

Integrated documentation for CVS, CSS, HTML, JavaScript, MySQL, PHP, PostgreSQL  and Smarty – very nice!

The project system works OK for PHP projects, but it is cumbersome and has odd limitations (you can’t copy and paste files and folders, for example). 

Non-standard Alt+Tab behavior is not very useful.

No Subversion integration

No integrated code reformatting (help provides a good explanation of how to integrate 3rd party tools, but these don’t seem to work well, either).

One of the real selling

The ability to edit files via FTP or SSH file transfer is nice.

Integrated support for SSH terminals is nice (but why, oh why is copy and paste so difficult?)

Other Suggestions

A lot of people use Dreamweaver to edit PHP code. I guess if you come from the design world, this makes sense. Syntax highlighting for PHP is pretty poor, but I will say that Dreamweaver has the best support for code reformatting.

Notepad++ has a good syntax highlighting of many languages and there are add-ons to support FTP and integrated spell check.

Here is an article I found interesting: http://politics.slashdot.org/article.pl?sid=07/02/25/1935215

Notice how the high brow, traditional designers favor fixed width layouts with lots of wasted space in the margins. Every single one of these sites is fixed width. Most have top and bottom margins – a waste of valuable screen real estate, IMO

So why do traditional designers favor these kinds of layouts? I’ve seen lots of arguments, some lame and some worthwhile, but who cares what the designers prefer? Most designers are as much a part of the “typical audience” as I am – no part at all. The job of the web site to please it’s audience, not it’s designers. That is easy to say, but how do we really figure out how to best please the “the audience”? It’s something I’m still working on. Listening carefully to the client is one idea – often they have spent years working with their clients. Over those years, they have built up instincts about what will and won’t work. For interactive sites, I’ve got a stack of books that explain, in painful detail, how to create user interfaces that function well. These books are backed up with actual studies of end user actives. But web sites need to be both functional and pleasing to the eye.

Another related thing I’ve realized: Designers prefer smallish fonts. I guess, to their tastes, small fonts look better in the layouts. However, over and over again I’ve found that most end users (well, the ones that actually read the content) prefer larger font sizes. If you ever want to be a hero for a day, find an office with lots of 17″ 1280×1024 LCD monitors and switch everybody from “Small Fonts” to “Large Fonts” (under the Windows, it’s under Advanced Display Properties). They’ll be cheering you as you walk out the door! Overall, I find flow layouts work better with larger font sizes. With a small font size, you can cram lots of stuff into small layout boxes. Flow layouts and larger fonts force you to work out other solutions. Flow layouts also allow you to support the brower’s ability to change font sizes (YUI fonts has nice support for this). This is a great way to easily accomidate low vision and older users.

Most designers probably don’t know how to make flow layouts, so they think they are too hard. Once you learn the skills, flow layouts really aren’t so much harder than the tricks you have to use to brow beat your HTML into a cross-browser fixed layout. They’ll also make translating your site into other languages a whole lot easier, if you think this might ever occur.

I’ve also read that fixed layouts have to be done to get content to fit properly into the boxes in the layouts. For busy sites, like the presidential candidate site home pages and sites with paid advertising, the need for a fixed layout makes some sense, but I’m not sure it’s something that can’t be worked around with a different design philosophy.  Advertisers expect their ads to appear in certain locations. However, Google, Slashdot and many other sites run on ad revenue have showed us that ads don’t have to be flash or bitmaps or be fixed at a pixel precise location. “We have to direct the reader’s attention” is a common cry. “Focus them on the buy button”, they say, as if a clever graphic design could really fool somebody into buying something. And as if amazon.com has not been showing us how to do this properly in a fluid design for the last 10 years.

Yes, a flow layout won’t look like a printed brochure, but so what? Only us geezers and designers that lived through the days of printed sales literature find comfort in layouts with portrait style proportions. Look at the proportions of your monitor – wider than tall, yes? New wide screen TVs and monitors are even wider. My teenage kids spend way more time looking at at the wide, fluid web pages of their favorite interactive web sites than narrow, glossy sales slicks. What do you think they will prefer when they grow up and get jobs (and the money and power to buy things)?

I’ve heard it said that columns of text need to be kept narrow because “wide columns of text are hard to read”. This is just a weak excuse. If I ever find a web page uncomfortable to read because it’s too wide, I’ll just make my browser window narrower. Power to the people, I guess. I’ve heard lame excuses about how users don’t know how to resize browser windows. Maybe users with 800×600 screens don’t know how, but flow layouts (and worries about too wide columns of text) aren’t for them, anyway.

When you design software user interfaces, you have to trade off between first time users and users of intermediate ability. Research shows, for a typical application, there will be far more users of intermediate ability (users that might, for example, know how to resize a window). But the proper audience to design for really depends on the application. For a presidential candidate web site, I could argue either way: Sure, you have the obvious sales brochure aspect of the site, but a good presidential candidate web site will truly be devoted to interactive activities such as volunteer sign up, forums and polls. For a web site that is just a slick brochure with limited interactivity, I can almost grasp the argument that users might not know how to resize their browser windows. Thus, we need to force a width to keep the columns comfortable to read. However, in the particular case of presidential web sites, aren’t the candidates hoping to draw you into the interactive part of their site? Interactive sites are much better with flow layouts. What type of layout is used behind the home page of your favorite interactive site?

A final consideration I’ll leave you with: What happens when your site is viewed on a device like the iPhone (or a Sony PS3, XBox360 or Nintendo Wii)? In the limited space available, it’s important to make the most of screen real estate. Only a fluid design can accomplish this (without having a site dedicated to each device). My brother has an iPhone. I was interested to find out that he uses it often to do the kind of web surfing that I associate with sitting at a desk. His iPhone lets him web surf in a lot of situations where I would be reading a book or magazine. It’s interesting to consider my web sites viewed this way. Fluid designs work now and have the benefit of being ready for the future. 

When I get a chance, I’ll post some pictures of flow sites that I’ve worked on running on the my daughter’s Wii and my brother’s iPhone.  They look great ( considering the limited resolution available).

Logging in Asp.Net

Logging in Asp.Net is confusing because Microsoft uses the word “trace” in two different contexts:

When a page is rendered and HTML generated, you can have the Asp.Net engine add a bunch of additional page state information to the generated HTML. In your application’s web.config file, this is the tracing that is controlled by
<trace></trace> under <web.config></web.config>. In practice, I don’t often find this kind of tracing helpful, but when I do,  I put the following tags in web.config to make it easy to turn on or off:

<system.web>
  <customErrors mode="Off"/>
  <trace enabled="false" requestLimit="10" pageOutput="true" traceMode="SortByTime" localOnly="true" />
  <!-- ... -->
</system.web>

With these tag in place, you can enable and disable page state tracing (more like a dump if you ask me) by changing enabled=”false” to enabled=”true”.

The other use of the word “trace” is related to the more traditional form of logging via .Net Debug.Write and Trace.Write. Essentially, under Asp.Net, you use Debug.Write and Trace.Write as usual. Contrary to the MSDN documentation, the default compiler options for Asp.Net web applications allow Debug.Write output in Debug builds and allow Trace.Write output in both Debug and Release builds. If you run your Asp.Net application under the Visual Studio debugger, you will see that Debug and Trace output is redirected to the debug output window without any additional configuration.  As with any use of .Net logging, you can setup tracing to various outputs either in code or via your application’s configuration file. To configure your Asp.Net web application to log trace output to a file (for use with other tools), place the following into your web.config:

<system.diagnostics>  
  <trace autoflush="true">    
  <listeners>    
  <add name="MyApplicationTracer" type="System.Diagnostics.TextWriterTraceListener, System, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" initializeData="MyApplicationTrace.log" />
  </listeners>
  </trace>
</system.diagnostics>

This will result in debug (in debug builds) and trace output going into a file named MyApplicaitonTrace.log in your web site’s root folder.

Logging in .Net Command Line Applications

Since .Net applications are easy to create and fairly expressive, I often write small command line utilities that, on other platforms, I might create as shell scripts.  Here are some handy code snips that I use to handle simple logging for these applications:

namespace Program1
{
 ///


 /// Prepends each output line with the time
 ///
 public class TextWriterTraceListenerWithTime : TextWriterTraceListener
 {
  public TextWriterTraceListenerWithTime()
  : base()
  {
  }public TextWriterTraceListenerWithTime(Stream stream)
  : base(stream)
  {
  }public TextWriterTraceListenerWithTime(string path)
  : base(path)
  {
  }public TextWriterTraceListenerWithTime(TextWriter writer)
  : base(writer)
  {
  }public TextWriterTraceListenerWithTime(Stream stream, string name)
  : base(stream, name)
  {
  }

• Simple Tracing and Logging for Asp.Net and .Net Command Line Applications

Wow, some programmers go crazy about logging.  Do a Google search for “.net logging” and you’ll see what I mean. Some of these libraries are huge.  While some applications may really need all the complexity these libraries offer, I have seen it severely abused on more than one occasion. Crank up a copy of Blackberry Enterprise Server (BES) and let it run for a week or two. You’ll find your Windows Event Viewer overflowing with all types of mysterious, confusing and generally useless messages.  In addition to the constant Event Viewer spew, if you let BES run unattended, you’ll eventually find that your BES hard drive is literally overflowing with many different log files all with obscure names and even more obscure contents.  Exchange Server has a similar problem where some logging is done in Event Viewer, other logging is done to text files. And none of the Exchange Server logging options ever seems to enable the simple email flow messages you need for common day to day administration tasks that I take for granted on our Linux email servers.

Is all this really necessary? After 20 years of programming, I’ve found that the following simple API is more than enough for all my needs:

LogMessage(componentName, severity, format[, formatArg1, formatArg2, ...])

Depending on your programming language and coding standards,  this simple function may need to be wrapped in a class. You might also wrap the function call in a macro to allow for conditional compilation and to keep from having to pass the same componentName over and over.

With this API in mind, you also need a fairly simple configuration system to automatically prefix the messages with time, date, and thread ID and direct messages to some combination of syslog, Windows Event Log, a text file (just 1, please!), console, system debugger, email (for Alerts – see below)  and user defined functions.

The componentName argument is best handled as a single string determined by the programmer. This keeps things simple, yet gives the programmer a lot of flexibility in both the presentation of messages and the ability to filter messages.

There is no sense in going overboard with adding lots of flexibility for the severity argument. It’s best to spell things out and keep them simple. You’re never going to get a group of programmers and system administrators to agree on the difference between “error”, “critical” and “alert”. I use an enumerated type with the following values:

• Debug – Low level messages of interest only to programmers. It should be easy to enable and disable output of debug messages. The only reason a user would enable debug messages is to create a log file to be sent to a programmer. Information messages may be sent to log files and/or UI elements, but they should never be sent to system facilities (such as syslog or the Windows Event Log).
• Information – Messages that let users know the program is doing what it supposed to do. Information messages are of most value for services where there is no interactive UI to demonstrate that the program is working properly. However, interactive programs can trap Information messages to display status messages during length operations. Information messages may be sent to log files and/or UI elements, but they should never be sent to system facilities.
• Warning – Messages to let the user know the program encountered a problem, but the problem is not completely unexpected. The program should be able to work around problems that result in warnings (possibly by retrying at a later time).  Warnings should be sent to all log files and system facilities.
• Error – Messages that need the user’s attention.  Errors should be sent to all log files and system facilities.
• Alert – Messages that need the user’s immediate attention.  The user should be woken from bed. Alerts should be sent to all log files and system facilities.

I’ve found that the most difficult choice is between Error and Alert. For example, at an ISP, email server problems are almost always Alert messages, while for home users or small businesses, email problem can wait until morning. If it’s not obvious, make everything an Error and assume the administrator will have some way to filter the messages to decide for himself which Errors are really Alerts.

The data type for the format argument depends on your programming language and I8N requirements and tools. For ease of programming, I much prefer to use String.Format style format strings (printf strings in C++ and PHP code).  

I could write an entire article about choosing the best method for formatting strings for output, but quickly:

For production applications, use formatting rather than composition:

• Do use: string output = string.Format(“A = ‘{0}’”, a)
• Do not use: string output = “A = ‘” +  a + “‘”;

As much as I love C++ and Bjarne Stroustrup (and understand the arguments for the ability to format user defined types in a safe and type safe manner), favoring formatting over composition leads to programs that are easier to write, easier to read and easier to translate.

If you need to translate your program into other languages, there are two schools of thought for handling literal strings that need to be translated:  

• In the Windows/Mac style system, each string is manually assigned a unique integer identifier and all strings are stored in a database of some kind (typically application resources). When the you need a string, you program a call to the database API, passing it the unique integer ID of the string to be retrieved as a lookup key. 
• In the gettext style system (UNIX like platforms),  translated versions of the strings are still stored in an external database (though typically in simple text files). However, the original strings created by the programmer (often English) remain in the code. The original strings are used as the lookup key to retrieve the translated string.

Kind of like the difference between using raw char* vs. a string class for low level string operations, the Windows style system is the most resource efficient. The gettext system is less tedious to program with and, for better or worse, more tolerant of errors. You can find tools to retrofit your old code for either style. Also, there are various kinds of tools for both systems that allow translators to translate application strings without access to program source code.

Since my API is a lowest common denominator, mapping my simple API to syslog and the Windows Event Log is fairly straightforward.

Eventually, my credit card processor teamed up with one of the first companies to offer an Internet credit card processing gateway, Cybercash. I hopped on board as one of the first customers. As with a lot of APIs created by hurried teams of inexperienced programmers, Cybercash was much more complex than was necessary to accomplish the simple task at hand. Credit card processing can be really simple: Pass the credit card processor a list of required fields and get back an Boolean success code and an error message.  HTTPS is a fine protocol for this type of stateless transaction that must occur over a secure channel. Input fields can be passed as a URL encoded query string. The error code and error message could be returned an HTTP response code or via a URL encoded fields in the response body. The modern, fancy name for this type of transaction protocol is REST. Under the hood of its overly complex API, Cybercash used the simple REST protocol I described earlier. Liked a lot of other Cybercash programmers, I eventually figured this out on my own and bypassed the complex and cumbersome API to use the simple REST protocol.

Time passed on and Verisign bought up all kinds of important technology related to e-commerce. They purchased Cybercash and started promoting their own gateway API, PayflowPro. Several times, we researched the fees and APIs offered by the various vendors. PayflowPro was the best and the one we used and recommended. The PayflowPro API was a tad bit easier to use than the official Cybercash API – easy enough that instead of bypassing the API for the REST protocol, I used the actual PayflowPro API function calls.

When the time came to automate the billing for T3city, I started out with PayflowPro. However when it came time to get a new merchant account, I took some time to review my options. On paper, PayPal Website Payments Pro looked pretty good – they have a nice and simple fee structure (unlike PayflowPro with all sorts of fees billed out at different times) and the fees are competitive if you can get your volume over $3,000/month (and not too bad if you can’t). The sign-up process was all online. I figured it couldn’t be too hard to change gateway APIs – I’d already used several others – so I made the switch.

Well, I was wrong about the difficulty level associated with switching from PayflowPro to PayPal. The PayPal API is much more complex. First, there is the fact that the API is SOAP based. In addition to the more complex SOAP protocol, the PayPal programming examples use another layer on top of the native SOAP API. Why they add this extra layer, I really don’t know – maybe it’s just a fact that every junior programmer feels like an extra layer or two will make things simpler. It doesn’t make the API any easier to use. In fact, I found the differences between the new layer and the documentation for the base SOAP protocol very confusing. In the end, I elected to ditch the PayPal examples and use PayPalSvc.wsdl and .Net’s SOAP support tools to generate my own SOAP client for the PayPal API. The resulting API is a little awkward is spots (mostly due to PalPals not so hot API design), but at least it matches the detailed and fairly comprehensive PayPal SOAP protocol documentation.

Once you get over the base complexity of the PayPal SOAP based API, then you have to deal with the stateful nature of PayPal. Credit card processing is normally a simple transaction, but the PayPal Website Payments Pro program requires that you offer your customers the option to pay via PayPal. And, what the heck, allowing payments by PayPal is not such a bad idea.  However, unlike a simple credit card transaction, payments with PayPal naturally flow over a series of pages, some on your site and some on PayPal’s site. Information such as billing and shipping addresses and invoice line items have to be exchanged in a delicate dance. Assuming you are willing to tackle the complexity of proper error handling, avoiding browser complaints about HTTPS redirects and giving the user the ability to cancel cleanly, getting PayPal transactions to flow smoothly turns out to be a fairly complex.

Is the extra complexity worth it? Consider these points:

• Both gateways are very reliable. 
• PayPal fee structure is simpler, clearly specified on their web site and may be less expensive – YMMV. Traditional store fronts may be able to get better rates from their current processors with a different gateway API.
• PayPal charges the same fees for all types of credit cards. AMEX fees are usually higher, so this may save you some money, but, again, YMMV.
• PayPal does not require any type of term or volume commitment
• Accepting PayPal is nice – giving customers more ways to pay you is a good thing!
• Implementing PayPal Website Payments Pro is a good deal harder than using PayflowPro. Depending on your web site/application, adding stateful credit card / PayPal transaction processing may require adding new database fields, new screens and other large scale programming changes.
• Under .Net, PayPal Website Payments Pro does not require any native (non .Net) code. To use PayflowPro in .Net, I load and make native function calls into the native PayflowPro DLL.
• PayPal Website Payments Pro requires full names for states (not the typical 2 letter codes) and proper ISO country codes for every transaction. If you don’t already have validation in place and tables to translate state codes into state names (users will use 2 letter state codes if you let them), this support will require additional programming work.
• With PayPal Website Payments Pro, you have to perform an additional step to transfer money from your PayPal account into your bank account.  Funds from credit card purchases show up in your PayPal account immediately, but transfers to your bank account take multiple business days (similar to the delay you’ll have with a traditional merchant service provider and PayflowPro). In theory, you can use funds in your PayPal account immediately for purchases via PayPal, but you would need a careful strategy to ensure proper accounting.
• To credit a customer via PayPal, you must create a new credit transaction that references a special code from the original sale transaction. Unlike PayflowPro, you cannot void a transaction and you cannot change a sale transaction to a credit transaction.
• PayPal has different and likely much more strict fraud protection measures. Overall, I like this capability, but it did cause some pain during our switch when credit cards on file from long term customers no longer worked because of mismatched a CVV number, a mistyped expiration date, etc..

Most of my projects have a readme.txt file. Typically, a readme file will include an overview of the project and, if it is a code library, a few key source code samples ready to copy and paste. In the past, I have considered using .html files or even .doc files instead of .txt files. I’ve stuck with readme.txt for two reasons: One, I’m mostly working in a programmer’s editor. True, many IDEs have built-in HTML editors, but they are usually slow to load. Two, all the worrying about fonts and styles can be distracting. Overall, both these problems introduce what I’m going to call friction for lack of a better term. When you’ve built up some coding momentum (and this is important to do for reasons I may one day get around to writing about), friction slows you down. Some types of friction are important to the overall goals of a project and just can’t be avoided, but if friction is avoidable, then I say avoid it and gain some coding efficiency. Over time, the overviews and code samples in a readme.txt file can be pretty important to a project.  HTML (or Word) formatting is pretty, but it increases friction for no practical benefit that I can see. Thus, readme.txt instead of readme.html.

So, returning to the subject of browsing a source repository. I started thinking that maybe readme.html was not such a bad idea after all. All you really need, is <h1>, <h2>, <p> and <pre> tags. How much hassle could that be? Tables can be handy, right? So, first I started thinking about getting a global style sheet in place, next I started thinking about creating a template I could use to create a new readme.html.  Then, I started to think about how much friction I was going to add to day-to-day development and what the benefits will be. It’s a tougher choice than it was in the past, but, for now, I’m sticking with readme.txt.