blogs.teztech.com

No items inserted
Uploaded at: July 27, 2009 6:03:44 PM CST
Processed at: July 27, 2009 6:09:30 PM CST

For more details on the status of your data feed, go to http://www.google.com/base/showfeedsummary?fid=1336756

When I click on the link, I get the same “8660 of 8660 items inserted – Processed on Jul 27, 2009 6:09 pm CST” page I’ve already seen.

No, we haven’t ever had any problems with disapproved items. There is one inactive/disapproved item right now – it’s the item with the title “Test Item 1″ I created on Friday to see if I could manually add an item.

Today, I fixed a problem with custom attribute names (we were including spaces). This problem was always there, but I figured fixing it was worth a try. My upload just finished processing and it hasn’t seemed to help… still no items.

As you suggested, I filled out a support request. Thanks.

I have been experiencing this issue for over 2 weeks now.  I have been in contact with someone named Atanu at Google, & have sent well over 7 replies to him.  He has been somewhat responsive but each time just responds with a canned response saying “our engineers are working on resolving this issue. We sincerely apologize for the inconvenience.”  As I’m sure is the case with you, this has had a significant impact on our traffic and sales.  Has anyone else who is experiencing this problem found a workaround or a more solid response from Google?

From the comments above, I suspect that the problem has something to do with the actual data in my data feed file. eimaj, maybe you can pinpoint the offending data via binary search: Divide the data in halves, then try to process each half separately. If one of the halves works, then divide the half that doesn’t work in halves (quarters of the original file) and try to process each new half separately and so on. This should at least be faster than trying 100 products at a time. Maybe something we added or updated around June 17 is causing the Google data feed file processing code to crash and/or reject all our items. Or maybe Google made some change to the code they use to process data feed files.

Here is the email I just got from today’s upload:

No items inserted
Uploaded at: July 28, 2009 7:37:05 AM CST
Processed at: July 28, 2009 8:10:40 AM CST

But, still, when I login to look at the data feed, I see “8660 of 8660 items inserted – Processed on Jul 28, 2009 8:10 am CST”.

This mismatch between the count of items inserted in the email and the web page is some kind of bug, but it doesn’t help point me to what is causing the problem. Just a little feedback from Google might go a long way…

Since others are reading this thread and having similar problems, I’m going to share some of my code and see if anybody else sees anything obviously wrong. Going with the theory that there is a problem in the data file, maybe there is a problem with the way I am encoding the file’s data fields. Here is the function I use to encode most data fields:

static char[] _quotedCharacters = new char[] { ‘\t’, ‘\r’, ‘\n’, ‘”‘ };

// GoogleBase specific rules
// See http://base.google.com/support/bin/answer.py?answer=25200&hl=en_US
public string TsvQuote(string s)
{
if (s.IndexOfAny(_quotedCharacters) >= 0)
s = “\”” + s.Replace(“\””, “\”\””) + “\””;

return s;
}

Here is the header line from my data feed file:

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string

We use our own categories for product_type. The way the values are input into the system, the users can’t enter in non-printable characters, so I just do a simple search and replace for double quotes:

StringBuilder categoryText = new StringBuilder(category.Name);

while (category.ParentCategoryID != null)
{
category = Category.FindCachedCategory(category.ParentCategoryID.Value);
if (category == null)
{
categoryValid = false;
return “”;
}

categoryText.Insert(0, category.Name.Replace(“\””, “”) + ” > “);
}

return categoryText.ToString();

All the above code seemed reasonable at the time and was working OK until about June 16. OpenOffice has no problem parsing my file properly. I do have some HTML embedded into descriptions, and I’m not sure how GoogleBase treats that, but the HTML is not new.

1) I am worried about product_type. The GoogleBase specifications aren’t crystal clear… I coded up my best guess and it seemed to work, but maybe it really doesn’t. Here are some examples from my data feed file:

“Clothing > Men’s Clothing > Men’s Lifestyle Clothing > Men’s Lifestyle Pants”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $50 – $74.99″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $75 – $100″,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”,”Gift Ideas > Graduation Gifts”
“Accessories > Gift Cards”,”Gift Ideas > Gifts $100+”
“Gear > Camping/Hiking > Backpacks > Fast and Light Packs”,”Gear > Camping/Hiking > Backpacks > Weekend Packs”
“Accessories > Book Bags”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Activities > Adventure Travel > Travel Gear > Travel Packs”,”Gear > Camping/Hiking > Backpacks > Day Packs”
“Gear > Camping/Hiking > Backpacks > Day Packs”,”Activities > Adventure Travel > Travel Gear > Travel Packs”
“Footwear > Men’s Footwear > Men’s Multisport Footwear”,”Activities > Climb > Climbing Shoes”,”Earth friendly Gear”
“Clearance > Winter Sale”,”Clothing > Kid’s Clothing > Kid’s Shirts”

As you can see, many of our products are in multiple categories / product_types and I believe we handle this correctly according to GoogleBase, but maybe somebody can see something I’m doing wrong.

2) I upload products with 0 quantity in stock. I know that these items are automatically deactivated, but that is what I want and it seems to be the most straightfoward way to make sure that products that go out of stock in our stores are removed/deactivated from GoogleBase.

278 of 291 items inserted
Uploaded at: July 28, 2009 11:33:31 AM CST
Processed at: July 28, 2009 11:46:58 AM CST

And now a couple of hours later, I finally have 273 items “Published… searchable soon”. Between character encoding errors in description or quantity = 0, the missing items can all be accounted for. So, the problem seems to be somewhere in the content of my other 8369 items… Item # 292 has some Word HTML garbage – my current theory is that the source of the problem is with the HTML in some of the descriptions. I have figured out that some HTML is OK, because tables and bullet lists seem to make it through. One idea is that if any of the descriptions contain certain tags (<head>, <title>, <style>, <script>, etc.), then some final stage of data feed processing crashes and the entire feed ends up in the bit bucket.

This morning, I got a fairly generic email response from GoogleBase Support:

Thank you for your email. I have reviewed your Google Base account and the
reason few items your file are failing is because your feed contains
unsupported characters. For example, certain white space characters, such
as tabs, may not process correctly and will result in a feed failure.

To correct this error, please remove all unsupported characters like HTML
tags etc from your feed and upload your file again.

Please let me know if you have any further questions or concerns.

Unfortunately, this response was not helpful. I already knew about the “few items” and, as mentioned here, I have submitted multiple data feed files that don’t have any of these issues, but still fail to update my GoogleBase items.

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
“<ul>
<li>A multipurpose rescue/rappel rope.</li>
<li>100% nylon low stretch static.</li>
<li>Uniform sheath coverage for durability and gear compatibility .</li>
<li>Ideal for heavy duty alpine, big wall, and expedition fixing and haul lines.</li>
<li>Great for mountain rescue and industrial rope access.</li>
<li>Certified to NFPA 1983, 2006 edition for Life Safety Rope forlight use.</li>
</ul>” 9995 http://www.mountainhighoutfitters.com/Products/SuperStatic+7%252f16%2522+-+300ft.html 255.00 “Sterling Rope SuperStatic 7/16″” – 300ft” Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-1.teztech.com/Products/7662.jpg 8.00 pounds 1 new

My current theory is that it is the URL that is tripping it up. This URL is valid (works in IE, FF and Chrome), but it does contain URL encoded characters ( / and ” ) that could cause problems if unescaped. The other interesting thing about this product is that it was added to our database 14 May 2007, so it was working OK at some point (we were able to upload all our products to GoogleBase successfully in the past).

description id upc link price title brand product_type image_link weight quantity condition color size c:frame_lens:string c:flavor:string c:length:string c:waist_x_inseam:string c:zipper:string
The Nose in a day, dogging your project for the painful 20th day, and everything in between, this 10.4mm rope handles all ends of the spectrum. Lightweight, it will never slow you down for a second and will always return day after day to handle the abuse your projects dish out. 39705 http://www.mountainhighoutfitters.com/Products/Marathon+Sport+70m.html 224.00 Sterling Rope Marathon Sport 70m – 70m Sterling Rope “””Gear > Climbing > Rope/Cord/Webbing””” http://mhoimages1-2.teztech.com/Products/7661.jpg 7.00 pounds 2 new 70m

There are no double quotes around the description because it shouldn’t need them.

I am going to submit this file again right now just to be sure.

PJ

Tab and comma delimited files existed long before Google decided to allow their use for data feed files. My understanding has always been that, in these types of files, a field “needs” to have quotes around it if it contains the delimiter character or line feed. Here is what Google has to say about this subject: http://base.google.com/support/bin/answer.py?hl=en_US&answer=25200. It is not explained how the use of quotes interacts with the  “Use Quoted Fields” setting, but I always set this to “Yes” under the assumption that “Yes” allows the use of quotes around fields (but does not require it). I have many successful runs that use the code I posted above, so I think, at least to some degree, GoogleBase also understands that fields within a tab delimited file may or may not have quotes around them, depending on if they are “needed” or not.

I am still working on this (and just hoping that maybe somebody else sees something obvious)… if I find out for sure that the data feed file I’m working on now with just a single product is causing the problem, then I will post the file for downloading. Here are the settings I use for all my data feeds:
Feed format: googlebase, File format: text, Encoding: autodetect (detected as UTF-8), Delimiter: tab, Use Quoted Fields: Yes

Where did you read that only one product_type value is supported in tab-delimited feeds? This page seems to suggest that you can have more than one: http://base.google.com/support/bin/answer.py?hl=en&answer=66818. Looking at the raw data feed file and some items that have gotten through, I figured out that I probably need to quote the comma separated product_type values individually rather than trying to quote the entire field (right now, all my product_type values end up with double-quotes around them). But this is a minor problem and I am failure sure it’s not the cause of whatever is keeping my big file from getting processed.

1 of 1 items inserted – Processed on Jul 31, 2009 7:20 am CST
Detected encoding: US-ASCII
Detected delimiter: Tab (\t)
Detected file format: Text
Feed Level Messages
Warning: Missing recommended attribute: upc
While items missing recommended attributes will process successfully, we recommend including relevant attributes if they are available.

And here is what the email from GoogleBase says:

No items inserted
Uploaded at: July 31, 2009 6:33:37 AM CST
Processed at: July 31, 2009 7:20:04 AM CST

The “No items inserted” email is how I determine that “the problem” has occurred. I have also noticed that it takes a relatively long time to process these files. I’m sure they go into a queue and the processing time depends on how busy the GoogleBase data feed servers are, etc., but I’ve been processing these files manually for 5 days in a row and I think an hour to process a 1 product data feed file is longer than usual.

The GoogleBase data feed documentation explains how to create tab delimited files by hand in Excel and OpenOffice Calc, so I would assume that, at least to some degree, the GoogleBase programmers had the output of these programs in mind when they created their import code.

Celebird, I agree that “length” probably shouldn’t be a custom attribute. Thanks for the second pair of eyes. I’m going to try making some changes to length and see what happens. Interestingly, the length in my data feed is the length of the product (a rope in this case) and not the length of the shipped package. I wonder which one Google means by “length”? If they can use length to calculate shipping charges, then it is the length of the shipped package. I’ll try and research this.

Thanks for the pointers to the product_type threads. I just received an email with “Our engineers are aware of this problem and are working to resolve it.” on my issue, so, unfortunately, this seems to be some kind of standard response.

Changing the name of my length field from c:length:string to c:product_length:string allowed the file with 1 product to be successfully processed. Now let’s see if I can process a real data feed file.

1. On the google base dashboard, click My items tab.  Then click Data feeds.  Next click the edit link adjacent to you data feed file so you can set or verify the base configuration.

2. You should be in the “Edit Data Feed” screen at this stage.  Verify the following (or adjust to have them)

File format:                     Text
Encoding:                       Autodetect
Delimiter:                        Tab
Use Quoted Fields:          No

Note, the last item should a No and NOT Yes in your case.  Then resubmit.  Hope this helps

Some might say that this was an obvious problem, but our Internet store allows our end users to create their own attribute groups (color, size, style, etc.). When we create a data feed file, there is some code that matches our attribute groups to standard Google attributes (color, size). If there is not a match, then we automatically create a custom attribute for GoogleBase. In our case, “length” in our data base is the product length, not the shipping length, so using a custom attribute actually make sense.

In addition, I ran into other issues that caused a simple problem like this to take an entire week to work out:

- Even though GoogleBase processing was failing, the data feed message on the web site said “8660 of 8660 items inserted”. The GoogleBase data feed processing code was crashing on my data feed file, but apparently GoogleBase doesn’t code in place to trap these types of errors and provide useful diagnostics to outsiders like me.
- In my 8660 product data feed file, only a few products caused the problem and there was no easy way to narrow down the source of the problem. I had to manually divide my file into parts, submit the parts manually, then wait on the “success” emails to arrive, then rinse and repeat.
- There was nobody at GoogleBase support with the willingness or ability (not sure which) to help me troubleshoot this issue.
- Documentation for data feeds is just vague and wrong enough to keep you guessing. The biggest documentation issue I see right now is with product_type and how it interacts with the Use Quoted Fields setting.

If you run into a similar problem, the first step is to make sure you are getting all the data feed emails. Only the email said “No items inserted”. Next, you must isolate the data that is causing the problem. I assume my problem with a custom attribute named “c:length:string” is pretty specific, but the technique I ended up using to isolate it is general (if slow and painful).