Low schema storage for business data

No doubt if you've careered in developing MRP/ERP apps. You've spent huge time trying not to change the data model, orchestrating development processes, and developing new features for customers only to find they won't upgrade because a data conversion is too disruptive.

Nowadays start-ups are using meta-data driven frameworks or MVC model driven entity frameworks to speedup the application development process. But have meta data driven apps become convoluted with data storage?

In the 1960's computers needed to be told what the schema was before data could be stored, there was insufficient processing power to examine the data for schema in real-time. Thus a 50 year period of high-schema business systems started. RDBMS and SQL standardized the data storage.

Business system customers and IT expect SQL storage today, but others like Google have innovated and moved data storeage forward to other practises. Proof is evidenced by all the action in the social internet space, informatics for life-sciences, distributed energy management.

A couple of interesting companies offering new ways of working. AsterData combines SQL with the first-ever In-Database MapReduce programming framework for high-performance mission-critical data warehousing. Cloudera provides the open source Hadoop framework, the basis for data management at many large consumer Internet companies.

Tossing around low schema business systems with peers, I'm generally told forgetaboutit; Transaction systems need highly schemad data to guarantee data consistency. But soonest a majority of business data storage will not be transactional, data will be text, pictures, video, audio, feeds. Meanwhile we keep using predictable schema to store unpredictable data. Gordan Bell and Jim Gemmell cover the shift to huge data in TotalRecall and project video will occupy about 50 percent of data storage.


For different ways of working with data. Marissa Meyer spoke at PARC on Innovation at Google: The physics of data. Brett Taylor on the How FriendFeed uses MySQL to store schema-less data.

Marrisa Meyer @ PARC: Innovation at Google: The Physics of Data
Brett Tayor @ CBS: How FriendFeed uses MySQL to store schema-less data

Is it time to dream differently about data storage for business?

Web services have failed SMBs?

Web services on premise requires skilled IT administration. Midsise to large business and of course the new 4 horsemen IT giants.



invlove lost of have many moving



http://www.supplychainstandard.com/liChannelID/25/Articles/2693/Enterprising+Technology.html

What Still Matters About Distance?

Working in a geographically distributed product teams is increasingly common. Real time web technologies help deal with the many challenges. In this video Gary and Judy Olsen of the University of California at Irvine talk about what still matters about distance and prospects for the future.

The video is long. Salient points below:

• No overlap in timezones, means teams members need to meet face to face. Learn about each other socially. Thereafter even phone calls are more productive.
• Work needs bundling into local autonomous development, going back and forth to a head office on decision making ruins productivity.
• Some timezone overlap allows clarification of questions.
• Many vendors sell tools for working in distributed geographies, few vendors provide eduction on how to use the tools for distributed working.

Internal design of Salesforce.com

Todd McKinnon, (ex) SVP for Software Development, Salesforce.com (now http://www.saasure.com/) gave a talk at Palo Alto Research Center. Searching the net my sense is Todd said more at PARC than at other forums. PARC is about sharing research conclusions, salient material is conveyed to get further understanding via Q&A from peers.

It's taken me about a year to grip the significance of the Salesforce.com architecture and the possible impact on social persistent shared business apps. Salesforce solved the problem of building CRM application Entry Forms from collections of private customer data in multi-tennant horizontal partioned web database.

Salesforce architected the fundamental architecture around the 2000 from lesson learned at CommerceOne long before the basis for scaling social web apps commonly became architecture patterns from search: Bigtable / MapReduce / Hadoop / Sharding. Thus Salesforce uses Orcale databases in a very unconventional pattern. Database on a Database and custom indexes (not Oracle indexs).

My conjecture: It may be possible to build multi-tenant business apps without strong schemas for certain shared business data by mining for key value pairs from transactional data written via entry forms in linear time and later mined for associated collections via a search interface.

Hybrid systems could emerge. ERP private business data in tradtional relational storage. (Write once update many times). And socially purposed shared data stored in a Hadoop pattern. (Write once / read many times).

Applications would need to be natural cases for sharing data such as Lot Serial Traceabilty across the Food Supply Chain.

The FDA and Public (why not, the public) could trace peanuts to farms, peanuts to contract mfgs, food products containing peanuts to distribution centers all the way to retail stores and your food snack. The FDA could probably improve traceabilty by sponsoring University/ACM research in this area.

Driving a bus between the details in this conjecture is easy, what I'm looking to find is grounds for innovation from new data patterns.

Compare the Salesforce metadata table schema to a map in HBase/Bigtable.

Force.com Multi-tenancy White Paper - via Todd McKinnon
Understanding HBase and BigTable - Jim R. Wilson
Todd's talk at PARC, excellent write up - Paul O'Rorke

Persistent Social Business Apps

A recap of videos covering the emergence of ubiquitous, persistent online identity, social graph, feeds and open APIs. Fundamentally these are transforming individuals behavior, expectations and experience across the Web.

Clara Shih, HCI graduate Stanford CS, creator of the first business app on Facebook, talks about her app and what the World Wide Web of people means for interaction, relationships and enterprise software design.



Alex Payne of Twitter talks about properly designing open APIs for interaction derived from simplicity, "explorability" and consistency.



The Gillmor Gang discuss social development derivative technologies location based APIs and predict 5-7 years of reckless innovation in Social productivity apps before they all come crashing down (APIs levering, API web services)



Fix Bayonets!

Larry Ellison Live

Larry Ellison.... Live and unscripted in conversation with Ed Zander at The Churchill Club, San Jose Sept 21, 2009.

Ed Zander choose Larry for the tonight’s gig, because he's still in the hunt, he started Oracle on $2000. Hit the wall in 1980's recovered and competed fiercely with Microsoft for leadership on SQL Relational Database performance. After consolidating the ERP applications space (PeopleSoft, Seibel, 30+ more) Oracle is now the 2nd largest software company in the world (an amazing record).

I learned Larry was head of engineering from day one and 32 years later, he’s still head of Engineering at Oracle today. From time to times he’s run other functions, but only until he found the right leaders to head up other functions. Under his leadership Oracle is now the World’s second largest ERP applications company. But as I learned at last week’s SAP World Day 2009 conference in same room in the same Fairmont, ERP has fallen off Moore’s Law.

SAP looked pretty much the same 10 years ago. The biggest innovation seemed to be Timeless Software, the ability of SAP version 7 to accept upgrade packs without ripping out and replacing the complete system. Exact Software’s SMB ERP apps have had Timeless Software for years (Arco van Nieuwland ACM 2005)

Meanwhile Google and the iPhone doing innovation in the consumer space have rapidly brought new innovations in productivity to people doing business bypassing the corp IT police. For example 8 major technology companies including Microsoft, Samsung, and Nokia are in the process of opening up App Stores to entice app development on their platforms. Roger McNamee a prominent Silicon Valley investor helped steer Palm to develop the Palm Pre identified as mega shift to the wearable PC mobility experience, and now business apps are running in the Cloud.

Knowing Larry Ellison’s passion for competitiveness, evidenced in his reputation for driving the top performing relational DB, the Americas Cup and Oracles ferocious reputation. Why does Larry buy SUN at this time in his career?

Yes - I was attending, to get insight into how Larry intends to steer Oracle to the kind of growth Google and Apple have experienced. Must be his goal - right!

Larry made the claim that Oracle regularly beats SAP but mostly admires Tom Watsons IBM. That after beating IBM at Software wants SUN to complete with IBM on hardware. He played both ends of the Cloud calling VC on Sandhill Road nitwits for not gripping that Cloud companies run like Google run on hardware and awesome engineering. That many visions for ERP in the Cloud are vapor; he ribbed Salesforce.com, saying there latest technology invention was Fuchsia. Also that Oracle can provision ERP on demand today – (in the Cloud, if you like).

I gathered than many people at the event also wanted to know why Oracle wants to acquire SUN. Based on the conversation with Larry and people at the dinner table, the reason is clear. To get ERP back on to Moore’s law by quietly making adjusts to the SUN technology stack and Java in particular (many of Oracle’s ERP apps, are Java apps).

For all the megalomaniac bravado and careful posturing, I don’t believe Larry said more than he needed to say. Further Larry announced Oracle just did a 5 year plan, and he will continue to head up engineering. My conjecture Oracle has 30-40 Java ERP apps many routed in Client Server d architecture, how better to move these apps to the next major architrecture more efficiently than by nurturing the Java / SUN technology stack.

I asked Larry an audience question about opening up the ERP BOM in the same way Google has opened up Google docs. He gave a sales marketing answer. That Oracle can server up ERP on demand today. Afterwards, I asked Ed Zander (Larry departed smartly). Ed said Larry was not going to publically answer my question. He needs to sell ERP licenses today. I think Oracle buying SUN starts a 5 year race. Who can transform ERP from flatline growth to Moores Law growth.

My conjecture is before Larry passes then baton, Oracle comes out with a transformational ERP suite in the Cloud.

Check back in 5 years?

Buchheit at Facebook