To Scrape or Not To Scrape?

We've been toying around with the idea of screen scraping for some time now. The main reason, wasn't to avoid entering a debate about whether it's fair or not, but rather because we weren't sure we even wanted to deal with hosting any content of our own.

When I say hosting content, I'm referring to large amounts of content (videos, photos, documents, etc). Not simple things like user information and a few photos. I'm talking YouTube. It reminds me a bit of the discussion that took place at my last startup. To hardware or not to hardware. Hardware is a beast I would avoid at all costs - unless of course that's your thing. We weren't a hardware company. Yet we forged ahead designing some hardware for our product (electronic ticketing). I didn't know that this was a problem at the time, nor was there much I could have done about it anyway. But an MBA and some perspective later... hardware was one of our downfalls. It was a money pit. The design, implementation, and rollout were all overly expensive. Especially when you have a company like IDEO design things. And we built this big bulky system. I mean, there were probably a few smart ways we could have made hardware work to our advantage. But we weren't hardware people - so we should have left it for them.

Back to Vyoo. We were playing around with hosting our own reviews and recommendations, photos, vidoes, etc. But then we realized that it wasn't our core competency. Why rebuild the wheel? So we decided to link out to other sites or to try to partner with sites in order to allow users to find this preexisting content. Well, it's hard to forge relationships before you have a finished product and we wanted to prove out our concept. So we decided to scrape some screens to pull some data into our site. We only provide snippets and we link out to the original sites. But it's a more user friendly experience for our users.

This decision made me think about the implications of screen scraping. Initially, I had been opposed to the idea. I felt like people were just duplicating sites. Where was the innovation, originality, or hard work? And in that sense, I still believe screen scraping is a poor choice for copycats. Bring something new to the table! Or use snippets of the data to provide enhanced value (Digg). But then how about user generated content? First of all, that's where I think startups can provide quite a bit of value - for the user's themselves. And after all, this is our information isn't it? If I leave profile information on Facebook. Well, that's my information. Facebook didn't go to Haas. It didn't take my picture for me. It only let me use their service. And it's getting something out of that by using that information (targeted ad revenue). If I put my clothes in a locker at the gym. Those are still my clothes. Sure the gym owns the locker, but they don't own my clothes. This is a much larger discussion - with lots of nuanced ways in which I would take a different approach to this question of ownership.

Anyway, that's where I stand. With permission (if it's user generated), sites should be able to leverage a user's information in a unique way. If it's not user generated, well, providing snippets and sending traffic to the original site is totally cool. Again, Digg comes to mind.

"Quick and Drrrrty" vs. "Doin it Right"

I've had some small wins this week. For those who are developers, you'll be able to relate. Those that aren't, well, you'll be able to relate too, just under different circumstances.

I'd been struggling all week getting something working for our site. Again, I'm using Rails, and while it's been a breath of fresh air compared to learning C back in college, I've still had my share of hurdles. See, I'm a perfectionist when it comes to my code. Sure, I can find a way to do something "quick and dirty", but I always like to write the best code possible. That way, if you never get a chance to go back and refactor (which often happens), it won't be the end of the world. So many people I know just code to get things out there. Then when it turns to shit, as it undoubtedly will, they spend too much time fixing what should have been done right the first time.

I'm aware that there are separate camps on this subject matter. And both have their advantages. For example, the "quick and dirty" method will get you out the door more quickly. Which has its advantages. So is there a compromise?

I realized this week that the answer to that question is "yes". I spent so much time trying to figure out the "rails" way to do what I was doing that I got bogged down in details. It just so happened to help me redesign our database so that the data and presentation layer fit more closely together. Which is one of the key components of Rails. But not being a pro left my presentation layer with too much code. I just couldn't figure out a better way to do this. So I left it... not feeling great about it.

But what I realized is that I now completely understand my data layer and am comfortable with its design. That part won't need to change. But now I can leave quick and dirty piece available for refactoring - which hopefully will happen. I've asked our other engineer to take a look when he gets time. I'm aware this might not happen... but it's more ok than an entirely dirty solution.

My point is that I think there can be a beneficial relationship between "quick and dirty" and "doin' it right the first time". Do right what needs to be right (the guts of the system, data, scaling, security, etc). And maybe, if you absolutely must, get that frontend component done quickly. Of course, there are exceptions. If an undue load is forced on your database because of your Q&D work....that's unacceptable. And if your system looks like shit... well, that's a problem too.

Jane Capital

My parents were in town this past weekend and my mom met her old college friend, Jane Lindner. It had been years since they had seen each other, but they quickly fell back into conversation like they were back in college.

I was excited to meet Jane. She's quite an accomplished San Franciscan, having spent many years in the Valley and across the globe working in venture capital, management consulting, and the clean tech sector. She started her own company back in 2001, Jane Capital, which focuses on corporate advisory and venture investments in the clean tech sector. She's been quite successful with her business.

We spoke at length about what we both do and she expressed sincere interest in helping us with our company. She's already offered to make some introductions for us.

Jane will likely be a valuable contact both with Vyoo and beyond. She's spent quite a bit of time working in APAC, a place I hope to move my career at some point down the road.

One of the lessons that I continue to profess is the value of cultivating relationships and your network. You never know who you're going to run into.

I Hate EBay! For Real!

(Note: Yea, my ebay handle Silentwarrior is totally lame. But I opened this account in '99. So give me a break)

I was all sorts of fired up today. Firing on all cylinders. I even woke up at 7. I never wake up at seven. I don't think I've seen the 7 hour from this side of night since '85. Yea, my sleep was crappy, but that wasn't from a lack of it. It had everything to do with a guest staying over who decided it was cool to turn the washing machine on at 4:45am. "I didn't think it would make any noise". Um. Ok. That guest is banned. For being an idiot.

I start checking my email and low and behold there's an urgent message from ebay: eBay Registration Suspension: NonPayment. Oh no, not this again. Talk about a time drain. It's not like I don't have anything else to do with my time.

As you know, someone hacked into my ebay account and listed items for sale. They sold a ton of them and accrued quite a large amount in listing fees. Ebay said they would cancel all the fees. They didn't. I received a bill. Spent hours talking to their shitty customer support via chat (they have NO phone support) and they told me next billing cycle, all would be handled. I assumed the ten hours I spent on their silly little chat customer service meant they got their story right.

Well, after that email this morning, I logged in, and my account was locked. Shut down. Incapacitated. Now I was livid. I had spent, literally, hours upon hours on the chat program several days ago getting this resolved. They had sent me back and forth from billing to fraud so many different times that my head is still spinning (in a chat window too! And every time I was shuttled back and forth, I was queued up again, having to wait to speak, er, chat, with the next available idiot). When I would begin to raise my voice to tell them they had been doing this 6 times already...you know, by CAPITALIZING WHAT I WAS SAYING, they wouldn't answer me and they would just send me back and forth between fraud and billing. It was a nightmare. That's when I was finally assured next billing cycle the charges would be dropped...no problem.

So today I get on chat to resolve this. I'm so angry I can barely type. This is such a nuisance I don't even know what to say about it. But it makes me realize when a company just doesn't get it. I write a lot about companies not getting it. And while in good times, this can be ok, because everybody is happily spending money, it is not a sustainable practice, either when the economy turns or when alternatives begin to pop up. With eBay, it'll be more about competition than about the economy. It will be about customer backlash and their own mistakes. A big error they already made was dropping 3.5 billion dollars for Skype. Man. That was dumb. But another is alienating customers. Last year they made some changes to feedback ratings and listing fees which put the eBay community on the offensive. Customers left in droves. They held boycotts (great story). I hate eBay so much right now. But I'm sure I'll still use them for things. At least until the next auction site starts competing.

The worst part of this whole thing is that I can't even complain to ebay. I've sent emails...I get a canned response back. There's no phone number. No management on the chat program you can talk to. It's a nightmare! And their partner company, PayPal, is no different.

Things are Rolling...

So this week is the first week in a while where I feel like things are really rolling again - full speed ahead. In the beginning of starting this venture, there were always things to be done which required my attention. Milestones I'll call them. But recently...there's only one really major milestone that I'm keeping my eyes on. And that's our alpha release. For a lookback, here are some of the milestones that kept me cranking away. These are mostly in order though some things certainly overlapped.

1) Agreeing to start this business. Check.
2) Researching our ideas. Check
3) Writing our business plan. Check
4) Getting some legal help. Check.
5) Getting people to help us. Check. Well, we're still finding people to help out in a different capacity. But the initial help was the hardest! Those psychologists were tough to find. Fortunately, we found some great people. Half check (since we still are looking for expertise all the time)
6) Designing and developing the website (ongoing)
6) Incorporating. Check.
7) Getting more people to help. Half check.
8) Entering the business plan competition. Check.
9) Losing the business plan competition. Check.
10) Redeveloping our approach. Check.
11) Starting some things over...website development, UI development... That piece sucked! But it's better recognize your problems ahead of time and making changes than being stubborn and stupid. Check.

Now starting over wasn't starting completely over. It was taking our learning from the previous 6 months and applying them towards a different and far better approach. And talking through our ideas with some people who were outside of our little bubble who gave us insight into how to approach this (Thanks Kenny C!).

When we began with our new approach... I realized that it was time to get involved in the development efforts. This was a slow and painful realization for me. I resisted. Afterall, I had gone to business school so I didn't have to code anymore. Now I had to learn a new language. I had to learn how to write Ruby (and Rails). And I had to realize that there were going to be some nights where I'd be coding again into the wee hours of the morning. But I succumbed.

I'm actually enjoying it. So far. Surprise surprise. I've been seeing a lot of progress on the development side - thanks to our other developer (Fajar) and my trusty and excellent UI guy (Abe) who have been helping. And I love RoR. And I see a light at the end of the tunnel. The product should be ready for some alpha testing come September... as long as our algorithm is complete by then. That's our next major milestone and the one that's finally going to put the last year of work to the test!

Looking forward to a busy few months...

The Algorithm

I had a nice chat today with Mark Moulton, a psychometrician here in the Bay Area. For a bit of a background, psychometricians are people who understand how to conduct analysis on data that's based on psychological measurements. It's actually a pretty small field. On one end of the spectrum, you have psychologists - and we all know there are plenty of them out there. Then you have statisticians, who are also quite common. What you don't have are a lot of people that understand the intricacies of the intersection between those two fields. Mark is one of those guys. We've spoken to quite a few statisticians, but one of the problems with the statisticians is that they didn't really understand what we were trying to accomplish. Sure they could look at the data and analyze it but they couldn't provide any insight or provide any sort of guidance in terms of how data collection was really setup.

We talked to Mark a bit about what we needed and what our end goals were. While he understood what we were up to, we're going to sit down on Wednesday to see if he'll be able to do some consulting for us.

That's one of the last pieces of the puzzle that we're putting together. The rest is just dealing with taking the time to build the product out. Which, I won't lie, is taking a hell of a lot longer than I had hoped.