Hadoop redundancy mgmt for Web Services

Today I was stumpedby a question, why is there a such focus on developing no FAIL web services, when by definition any highly distributed computing model is inherently unreliable.

Examples of talks on reliability in the Ruby camp How Not To FAIL At Web Services in the Microsoft camp, when services do FAIL how to debug

The issue of reliability seems likely to increase, StikeIron now has 50 webservices StrikeIron Services List We know we can expect many more services to emerge. How will we handle in a scalable simple way like Hadoop?



Consider a conceptual analogy Hadoop provides redundancy for data storage by providing software to manage stripping of data across multiple commodity data servers, known as sharding. The expensive forerunner was a single beefy server equiped with a hardware based raid array.

Now consider the options for redundant Web Services in Microsoft architected solution. The recommended recourse is to use a single instance of SQL Server with the SQL Server Service Broker to manage queues and re-queuing for failed webservices.

But the characteristics of a single instance of SQL server match those of hardware raid. Unplug the power supply and the system FAILS. Whereas Hadoop is designed for server failures. If a PSU a disk drive or complete server goes out, Hadoops software reassigns one of the 2 sharded data stripes as the new primary stripe and automatically handles data replication to at least 3 servers.

What would a software based solution for managing web services failures look like? Lets assume redundancy features similar to Hadoop. 3 message web services queues are maintained. If queue 1 blows, switch to queue 2, and replicate the lost queue. The solution can alos learn the pattern of availability through statistical trail and error, and react by calleing web services at reliable times of day. The management of web services can simply retry, skip, queue, re-queue or email the administrator like Hadoop does.

Microsoft is no sleeper and the .NET Services Bus, part of Azure includes message handing and for sure over time mechanisms for handing failed Webservices via Azure will emerge. But what about business architects who wish to implement independent redundancy of web services now.

Do you know of an Hadoop like mgmt infrastructure project or product for managing queues to external Web Services?

Some brain storm ideas
The .NET Service Bus provides Frictionless connectivity across applications via Azure. “Web services are redundant as we know them - Juval Lowy”

Friendfeed. Adding indexes to a database with more than 10 – 20 million rows completely locks the database for hours at a time. After some deliberation, FriendFeed decided to implement a "schema-less" storage system on top of MySQL rather than use a completely new storage system. how-friendfeed-uses-mysql-to-store-schema-less-data Deploys Memchached

Website login without password tears

As our lives increasingly revolve around web apps, we end up with multiple user-name passwords combinations. Because websites have slightly different password rules, or because I don't trust the site with my 'main' password, I ended up with many sightly different passwords. So many, my head was ready to explode.

I'd signed up for OpenID a couple of years ago. But few sites support OpenID, I wasn't using personally until a fellow researcher started checking out Microsoft Geneva, which looks awesome. But I need a standards based solution I can work with right now, and incorporate into single sign-on systems from SoftieGoogooYahoo in the future.

I've trusted VeriSign with https certificates and visited their HQ in Mt View for the Java Posse meet-up and was awed by the level of security (like entering a military base). Long story short, VeriSigns PIP, personal identity portal works as a secure relay leaving you with just a single 3 way secure OpenID to sign-in to almost any real 'login' web site.

check it out https://pip.verisignlabs.com/ slide show 'inspiration':

sharding

http://highscalability.com/unorthodox-approach-database-design-coming-shard



MDV & BASES presents:



"How to Scale a Startup Successfully: A Discussion with Two Silicon

Valley Startup CTOs"



Date: Wednesday, October 3

Time: 6:00 PM to 7:00 PM

Location: David Packard Electrical Engineering Auditorium,

Stanford University



Food and drinks provided following the talks!



Discussion topics:



1) Scaling to the Masses - How hi5 built an information architecture

to support millions of users



Speaker: Akash Garg, CTO and Co-Founder, hi5 Networks

(http://www.hi5.com)



Come hear about hi5's evolution and the tools and techniques that

have successfully allowed them to support the needs of 70 million

users. Akash will also discuss challenges faced and lessons learned

as hi5 scaled over the past 4 years.



2) Building scalable web services with practical tools



Speaker: Nathan Schmidt, CTO of PBwiki (http://www.pbwiki.com)



PBwiki has exploited Free and Open Source tools to build out a

scalable, performant infrastructure with very small capital

investment. The last few years have seen a dramatic change in core

pieces available to system architects, and we'll talk about some of

the practical approaches to horizontal scalability, availability and

performance for today's nimble web startup.

Windows System/36

Microsoft San Francisco hosted Startup Weekend San Francisco. Tickets for the Sunday Night Demos were $5.99, less than a matinee movie, with a Tandoori dinner included. Turns out, I was in for helluva treat. About 140 people, developers, designers, had ozmosed into 20 teams following un-conference format.

Unconference implies no rules, wrong!

Teams group together based on similarity of interest, merit skills, capability to-do the job and good spirit. The Startup Weekend goal is to develop a working demo (not power-point), and a presentation pitch to catch the interest of venture capital.

The 20 teams leaned young and no team had developed Windows apps, all demos were Web and mobile apps. I feel we are at the inflection point where the new generation app patterns offer a 10x increase in productivity over the prior.

Analogy, IBM System/36 apps were replaced by Microsoft Windows apps. Azure looks likes Microsoft’s next generation, and I expect Startup Weekend like events to evangelize and win mindshare.

Enough yap, the Sunday Night Demos!

Startup Weekend Sunday Night Demos 1-2

Startup Weekend Sunday Night Demos 2-2

Web 2.0 expo hall, takeaway

* Platform connections companies turned out (Facebook, IBM, MSFT, Salesforce)
* Search engines did not (Google, Yahoo)
* Private community platforms, showed up
* Consumer internet communities, did not (MySpace, Friendster)
* Specialist SaaS providiers turned out (Meetup)

Conclusions:
* Laser focused specialists grow, by squeezing out multi-players
* Platforms focus on niche, but generally add value by aggregating others content
* Key to growth focus on disaggregating own content, to aggregate others content

web 2.0 expo hall pics

View more presentations from Clive Boulton.

Microsoft loves Hadoop, do you?

Search Engines, spot the odd man out...

• Google
• Yahoo
• Microsoft
• A9

Increasingly Google dominates search, certainly it works really well and fast. Try Yahoo, it works really well now, they spent millions rewriting their search to use Hadoop.

Corporate search behind the file wall, how does your search perform..?

How many of use corporate search built using Microsoft IT Architecture patterns. What are these architecture patterns, a single 64-bit SQL Server, 4 replicated SQL servers? Really these patterns are stepped in Windows client server technologies, not high performance computing.

Maybe it's all about to change, Microsoft also loves HPC and Hadoop!

Microsoft purchased Powerset in June 2008 for $100 million, to help it catch up with the Hadoop based search engine companies. ZDnet reports Yahoo has over 10,000 Linux servers. Google is rumored to have many more servers. But how can ISVs bennefit from Hadoop?

In a couple of ways, Hadoop technologies are likely to be interwoven in Microsoft Azure services through Powersets natural language search built on Hadoop and Mapreduce. Microsoft even changed the internal policys, to allow the Powerset dev team to continue contibuting back to the Hadoop open source project. If not Hadoop then Microsoft research has their own Dryad approach (pointer c/o David Chappell)














Another way to access Hadoop rleated technlogies, a new company Cloudera staffed by rockstar developers is set to commericalise open source Hadoop.

Marten Mickos, sold MySQL to SUN for $1B, he's now an investor in Cloudera. Seeing another growth story?

Some ISV's will use Cloud services, another set will build out own data centers?

So, how many data servers will you have in 5 years?

destinationCRM.com: Lithium Diagnoses the Community

destinationCRM.com: Lithium Diagnoses the Community

Posted using ShareThis