Google GAE Technical Points

0. Google GAE Architecture

GAE Physical Architecture and Beneficials, Global Cache

Google Components Combination, GAE + Compute Engines + Services

1. SQL or NoSQL: Google App Engine DataStore

Quote from Objectify:
https://code.google.com/p/objectify-appengine/wiki/Concepts?tm=6

Since the datastore is conceptually a HashMap of keys to entities, and an entity is conceptually a HashMap of name/value pairs, your mental model of the datastore should be a HashMap of HashMaps!

When you save() your entity, it gets stored somewhere in a gigantic farm of thousands of machines. In order to perform an atomic transaction, the datastore requires that all the data that is a part of that atomic transaction live on the same server. To give you some control over where your data is stored, the datastore has the concept of an entity group.

Remember the parent that is part of a Key? If an entity has a parent, it belongs to the same entity group as its parent. If an entity does not have a parent, it is the "root" of an entity group, and may be physically located anywhere in the cluster. All its child entities (and children of those children) are part of the same entity group and colocated in the same part of the datastore.

Note that an entity group is not defined by a class, but by an instance! Each instance of an entity class which has a null (or absent) parent defines the root of a separate entity group.

Why not store all your data with a common parent, putting it all in a single entity group? You can, but it's a bad idea. Google limits the number of requests per second that can be served by a particular entity group.

It is worth mentioning that the term parent is somewhat misleading. There is no "cascading delete" in the datastore; if you delete the parent entity it will NOT delete the child entities. For that matter, you can create child entites with a parent Key (or any other key as a member field) that points to a nonexistant entity! Parent is only important in that it defines entity groups; if you do not need transactions across several entities, you may wish to use a normal nonparent key relationship - even if the entities have a conceptual parent-child relationship.

Transaction Limitations

When you execute a datastore operation, you will either be in a transaction or you will not. If you execute within a transaction:

• Each EG you touch via get/put/delete/query enlists that EG in the transaction.
• You can enlist up to 5 EGs, but single-EG transactions are fastest
• Queries must include an ancestor which defines the EG in which to search. You cannot query across EGs at all, not even in XG transactions.
• There are some quirks at the low-level API: For example, get()s and query()s will see the datastore "frozen in time" and will not reflect updates even within the transaction. Objectify hides this behavior from you; subsequent fetches will see the same data seen (or updated) previously.

Transactionless

If you operate on the datastore without an explicit transaction, each datastore operation is treated like a separate little transaction which is retried separately. Note that batch operations are not transactional; each entity saved in a batch put() is effectively its own little transaction. If you perform a batch save entities outside of a transaction, it is possible for some to succeed and some to fail. An exception from the datastore does not indicate that all operations failed.

Original Author: Nikolas Goebel
http://www.sitepoint.com/sql-or-nosql-google-app-engine-part-2/

When to choose NoSQL

Google Datastore is to store the "dumb" data. It is not RMDS. It mainly addresses the huge data store/query.

1. Can a single server provide the performance we need? Maybe by utilizing caching? In this case RDBMS is the way to go. Look for example into CloudSQL, AppEngine’s purely relational offering.
2. Do you plan on growing a multi-application environment working on the same dataset? Depending on your volume, RDBMS might be what you need, because it separates the database layer very strictly from the application.
3. Do you need Ad-Hoc queries? In terms of query flexibility, SQL is the clear winner.
4. Do you require perfect consistency? Even though there are ways to achieve strong consistency in the Datastore, it’s not what it was designed for. Again, RDBMS is the better choice.
5. Are you expecting millions of reads & writes per second? The Datastore provides automatic scaling to infinity and beyond, and it’s right there for you to use with AppEngine.
6. Do you need a simple, scalable way to persist entities with variable attributes? Even though you’ll have to handle consistency and data aggregation yourself, the schemaless Datastore should be what you need. And it’s integrated right into AppEngine, so it’s the ideal choice for quick prototypes with changing entities.

Continuous Query A.K.A Update Notification

It also support the notification when data is updated, Prospective search. The GAE App then can monitor the data change and notify the client side accordingly.  

Performance Tips

1. User Cursor instead of "LIMIT : OFFSET" to get the limit rows, such as second 10 rows

2. Only select the necessary fields by using projection query

3. Use "GET" instead of "Fetch"

Datastore

Datastore uses 6 Bigtables to manage the data. refer to
https://developers.google.com/appengine/articles/storage_breakdown for detail

• Entities table

This one Bigtable holds all entities for all App Engine applications

• Index tables
• EntitiesByKind 
• EntitiesByProperty ASC
• EntitiesByProperty DESC
• EntitiesByCompositeProperty 
• Custom indexes table

• Id sequences table
• ID sequences are used to generate numerical datastore IDs for both entities and custom indexes.

Blobstore

Insert / Retrieve / Edit, in bulk – Flexible

• Direct access to Blob data in memory – Fast access to Blob data
• 5MB in ~2

2. CloudSQL - RMDS, MySQL

MySQL isn't mixed with big data. When mentioned the big data, the scale is billions rows. In another words, if the data is billions rows, the first choice should be DataStore, not CloudSQL.

3. CloudStorage - File Storage

4. Task Queues - Asynchronies execution

There are 2 types, Push & Pull. Push is more like a asynchronies thread. Pull can allows you to execute the task on-premise, out of Google APE. 

5. Memcache

it is a another server. Each memcache access will involves network call. The normal latency will be 2 ~ 5 ms.

6. Instance Cache A.K.A Global Variable

It is much faster than the Memcache. It is GAE memory and same life span as the instance.

7. GAE Backend Instance

Configuration Limits

• app: 5 backends
• app: 10GB of backends
• backend: 20 instances
• backend: 10GB
• 10GB combinations
• B8x10
• B4x20
• B8x5 + B4x10
• B8x5 + B4x5 + B2x10

API deadlines apply

• urlfetch: 5s default, up to 10s 
• datastore: 30s

Size limits apply

• HTTP: 32MB requests
• urlfetch: 1MB request, 32MB response
• memcache: 1MB objects
• Blobstore: 2GB objects, 1MB response
• Mail: 10MB send/receive
• Tasks: 100KB

No uptime guarantee

• best-effort service
• expect polite and hard shutdown
• various causes
• Examples
• software bugs
• hardware failures
• emergencies

8. Socket API

Now GAE App can use socket API to send the message out, such as Apple Pushing Notification. So GAE App can send out the pushing notice to the 2 most popular platforms, iOS & Android.

9. MapReduce

Map

• Input: user data
• Output: (key, value) pairs
• User code

Shuffle

Collates value with the same key

• Input: (key, value) pairs
• Output: (key, [value]) pairs
• No user code

Reduce

• Input: (key, [value]) pairs
• Output: user data
• User code

10. Channel & Feed API

Channel API is to push from App Engine to browser. Feed API is to push internet feeds to the browser through "PubSubHubbub".

11. Tools / Librarys

Memcache/DataStore

https://code.google.com/p/objectify-appengine/

CloudPlatform Excamples

https://github.com/GoogleCloudPlatform  

gwt-dispatch

https://code.google.com/p/gwt-dispatch/

CloudPlatform Excamples

https://github.com/GoogleCloudPlatform  

Protocol buffer

https://code.google.com/p/protobuf/

Bigtable

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/bigtable-osdi06.pdf

Struts2 Tech Points

1. Struts2 Architecture

http://struts.apache.org/release/2.0.x/docs/architecture.html

老兵永远不会死，他们只是悄然隐去 - 缅怀星爷的经典《大话西游》

怀着爆棚的期待看完了我心目中的经典《大话西游》的续集《西游降魔》。却发现自己是不是真的是进入恐龙时代了，与现在的年轻人掉钩了。实在是看不出来这个片子怎么可能如此的超人气。满片尽是为了无厘头而无厘头的对白。人物为了搞笑而搞笑的出场安排。与我心目中的经典实在是差太远。不知道是不是因为经典的发酵而导致了这个实在是不怎么样的片子高票房，原本我也是准备一定要进影院看看这个后续是如何延续传奇的。现在看看，还好没有去。江郎既已才尽，还是将自己的经典定格吧！为何要狗尾续貂。

人始终要老去的。。。

How to Design App From Apple Developer Library

Design Your App with Care

If you are a new to developing iOS apps, you might be wondering where the app development process starts. After devising your initial idea for an app, you need to turn that idea into an action plan for implementing your app. From a design perspective, you need to make some high-level decisions about the best course of action for implementing your ideas. You can then proceed with developing your app.

iOS App Programming Guide explains in detail many of the concepts, architectures, and techniques mentioned in this article.

Do Your Initial Design

There are many ways to design an app, and many of the best approaches do not involve writing any code. A great app starts with a great idea that you then expand into a more full-featured product description. Early in the design phase, it helps to understand just what you want your app to do. Write down the set of high-level features that would be required to implement your idea. Prioritize those features based on what you think your users will need. Do a little research into iOS itself so that you understand its capabilities and how you might be able to use them to achieve your goals. Sketch out some rough interface designs on paper to visualize how your app might look.

The goal of your initial design is to answer some very important questions about your app. The set of features and the rough design of your interface help you think about what will be required later when you start writing code. At some point, you need to translate the information displayed by your app into a set of data objects. Similarly, the look of your app has an overwhelming influence on the choices you must make when implementing your user interface code. Doing your initial design on paper (rather than on the computer) gives you the freedom to come up with answers that are not limited by what is easy to do.

Of course, the most important thing you can do before starting your design is read iOS Human Interface Guidelines. That book describes several strategies for doing your initial design. It also offers tips and guidance about how to create apps that work well in iOS. iOS Technology Overview describes the capabilities of iOS and how you might use those capabilities to achieve your design goals.

Translate Your Initial Design into an Action Plan

iOS assumes that all apps are built using the Model-View-Controller design pattern. Therefore, the first steps you can take toward achieving this goal are to choose approaches for the data and view portions of your app.

• Choose a basic approach for your data model:
  • Existing data model code—If you already have data model code written in a C-based language, you can integrate that code directly into your iOS apps. Because iOS apps are written in Objective-C, they work just fine with code written in other C-based languages. Of course, there is also a benefit to writing an Objective-C wrapper for any code that is not Objective-C.
  • Custom objects data model—A custom object typically combines some simple data (strings, numbers, dates, URLs, and so on) with the business logic needed to manage that data and ensure its consistency. Custom objects can store a combination of scalar values and pointers to other objects. For example, the Foundation framework defines classes for many simple data types and for storing collections of other objects. These classes make it much easier to define your own custom objects.
  • Structured data model—If your data is highly structured—that is, it lends itself to storage in a database—use Core Data (or SQLite) to store the data. Core Data provides a simple object-oriented model for managing your structured data. It also provides built-in support for some advanced features like undo and iCloud. (SQLite files cannot be used in conjunction with iCloud.)
• Decide whether you need support for documents:
  The job of a document is to manage your app’s in-memory data model objects and coordinate the storage of that data in a corresponding file (or set of files) on disk. Documents normally connote files that the user created but apps can use documents to manage files that are not user facing too. One big advantage of using documents is that the UIDocument class makes interacting with iCloud and the local file system much simpler. For apps that use Core Data to store their content, the UIManagedDocument class provides similar support.
• Choose an approach for your user interface:
  • Building block approach—The easiest way to create your user interface is to assemble it using existing view objects. Views represent visual elements such as tables, buttons, text fields, and so on. You use many views as-is but you can also customize the appearance and behavior of standard views as needed to meet your needs. You can also implement new visual elements using custom views and mix those views freely with the standard views in your interface. The advantages of views are that they provide a consistent user experience and they allow you to define complex interfaces quickly and with relatively little code.
  • OpenGL ES–based approach—If your app requires frequent screen updates or sophisticated rendering, you probably need to draw that content directly using OpenGL ES. The main use of OpenGL ES is for games and apps that rely heavily on sophisticated graphics and therefore need the best performance possible.

Start the App Creation Process

After you formulate your action plan, it is time to start coding. If you are new to writing iOS apps, it is good to take some time to explore the initial Xcode templates that are provided for development. These templates greatly simplify the work you have to do and make it possible to have an app up and running in minutes. These templates also allow you to customize your initial project to support your specific needs more precisely. To that end, when creating your Xcode project, you should already have answers to the following questions in mind:

• What is the basic interface style of your app? Different types of app require different sets of initial views and view controllers. Knowing how you plan to organize your user interface lets you select an initial project template that is most suited to your needs. You can always change your user interface later, but choosing the most appropriate template first makes starting your project much easier.
• Do you want to create a universal app or one targeted specifically for iPad or iPhone? Creating a universal app requires specifying different sets of views and view controllers for iPad and iPhone and dynamically selecting the appropriate set at runtime. Universal apps are preferred because they support more iOS devices but do require you to factor your code better for each platform.
• Do you want your app to use storyboards? Storyboards simplify the design process by showing both the views and view controllers of your user interface and the transitions between them. Storyboards are supported in iOS 5 and later and are enabled by default for new projects. If your app must run on earlier versions of iOS, though, you cannot use storyboards and should continue to use nib files.
• Do you want to use Core Data for your data model? Some types of apps lend themselves naturally to a structured data model, which makes them ideal candidates for using Core Data.

After you install Xcode, configure your iOS development team, and create an app project in Xcode, you can start developing your app. The following phases of app development are common:

1. Start writing your app’s primary code.
   For new apps, you probably want to start creating the classes associated with your app’s data model first. These classes usually have no dependencies on other parts of your app and should be something you can work on initially. You might also want to start playing around with designs for your user interface by adding views to your main storyboard or nib file. From these views, you can also start identifying the places in your code where you need to respond to interface-related changes. If your app supports iCloud, you should incorporate support for iCloud into your classes at an early stage.
2. Add support for app state changes.
   In iOS, the state of an app determines what it is allowed to do and when. App states are managed by high-level objects in your app but can affect many other objects as well, therefore, you need to consider how the current app state affects your data model and view code and update that code appropriately.
3. Create the resources needed to support your app.
   Apps submitted to the App Store are expected to have specific resources such as icons and launch images to make the overall user experience better. Well-factored apps also make heavy use of resource files to keep their code separate from the data that code manipulates. This factoring makes it much easier to localize your app, tweak its appearance, and perform other tasks without rewriting any code.
4. As needed, implement any app-specific behaviors that are relevant for your app.
   There are many ways to modify the way your app launches or interacts with the system. For example, you might want to implement local notifications for a certain feature.
5. Add the advanced features that make your app unique.
   iOS includes many other frameworks for managing multimedia, advanced rendering, game content, maps, contacts, location tracking, and many other advanced features. iOS Technology Overview gives an overview of the frameworks and features you can incorporate into your apps.
6. Do some basic performance tuning for your app.
   All iOS apps should be tuned for the best possible performance. Tuned apps run faster but also use system resources, such as memory and battery life, more efficiently.
7. Iterate.
   App development is an iterative process. As you add new features, you might need to revisit some or all of the preceding steps to make adjustments to your existing code.

iOS UI Design Guid from Apple Developer Library

Design with the User in Mind

The success of an iOS app depends largely on the quality of its user interface. If users don’t find an app attractive and easy to use, even the fastest, most powerful, most full-featured app can languish in the App Store.

There are many ways to get from initial inspiration to popular app, and there is no single path that guarantees success. But there is one directive on which all successful app development depends: Design with the user in mind. The strategies and best practices summarized below are all based on this directive, and they represent a few of the many principles and guidelines you need to follow as you design an app. When you’re ready to get to work, be sure to read iOS Human Interface Guidelines for the complete story.

Understand How People Use Their Devices

If you're new to iOS, your first step is to become an iOS user yourself. Then, as much as possible, explore the characteristics of the iOS platform as a user, not as a developer. Whether you've used iOS-based devices from the beginning or you've never held one before, take the time to articulate your expectations and analyze your actions as you use the device.

For example, consider how the following device and software features affect the user experience:

• iPhone, iPad, and iPod touch are handheld devices that enable and encourage people to use them on the go. People expect apps to start quickly and be easy to use in a wide variety of environments.
• On all iOS-based devices, the display is paramount, regardless of its size. The comparatively small device margin around the display becomes almost invisible while people are engaged in using apps.
• The Multi-Touch interface allows people to manipulate content without the intervention of another device, such as a mouse. People tend to feel more in control of the app experience because they can use touch to manipulate onscreen elements.
• Only one app at a time is frontmost. Users can use the multitasking bar to switch between apps quickly and easily, but the experience differs from seeing multiple apps open simultaneously on a computer display.
• In general, apps don’t open separate windows at the same time. Instead, users transition between screens of content, each of which can contain multiple views.
• The built-in Settings app contains user preferences for both the device and some of the apps that run on it. To open Settings, users must switch away from the app they’re currently using, so these preferences are of the “set once and rarely change” variety. Most apps can avoid adding preferences to Settings and can instead allow people to make choices within the app’s main user interface.

Remember that a great iOS app embraces the platform it runs on and provides an experience that seamlessly integrates device and platform features.

Learn the Fundamental Human Interface Principles

As a user, you notice when an app makes it hard to tell whether it received your input, or when a popover seems to emerge from different areas on the screen without apparent reason. In cases like these, what you notice is the app’s failure to follow the fundamental principles of human interface design.

In this context, the term human interface refers to the interaction between people and devices, including the software that runs on them. An app (or a device) is easy and enjoyable for people to use when its human interface builds on the ways in which people actually think and behave.

Apple’s human interface design principles codify several high-level aspects of human-device interaction that have a perceptible effect on the user experience. As you design your app, keep in mind the following principles of HI design:

• Aesthetic integrity. Aesthetic integrity is not a measure of how beautiful an app is; rather, it’s a measure of how well the appearance of the app integrates with its function.
• Consistency. Consistency in the interface allows people to transfer their knowledge and skills from one app to another. Ideally, an app is consistent with the iOS standards, within itself, and with earlier versions of itself.
• Direct manipulation. When people directly manipulate onscreen objects instead of using separate controls to manipulate them, they're more engaged with the task and they more readily understand the results of their actions.
• Feedback. Feedback acknowledges people’s actions and assures them that processing is occurring. For example, people expect immediate feedback when they operate a control, and they appreciate status updates during lengthy operations.
• Metaphors. When virtual objects and actions in an app are metaphors for objects and actions in the real world, users quickly grasp how to use the app. The most appropriate metaphors suggest a usage or experience without enforcing the limitations of the real-world object or action on which they’re based.
• User control. Although an app can suggest a course of action or warn about dangerous consequences, it’s usually a mistake for the app to take decision-making away from the user. The best apps find the correct balance between giving people the capabilities they need and helping them avoid dangerous outcomes.

Follow the Guidelines

iOS Human Interface Guidelines guidelines that range from user experience recommendations to specific rules that govern the usage of iOS technologies and onscreen elements. This section does not function as a summary of iOS Human Interface Guidelines; rather, it gives you a taste of the types of guidelines that help you design a successful app.

Great iOS apps give people streamlined access to the content they care about. To do this, these apps incorporate user experience guidelines such as these:

• Focus on the primary task.
• Make usage easy and obvious.
• Use user-centric terminology.
• Make targets fingertip-size.
• De-emphasize settings.
• Use user interface (UI) elements consistently.
• Use subtle animation to communicate.
• Ask people to save only when necessary.

Users expect apps to incorporate platform features such as multitasking, iCloud, VoiceOver, and printing. Even though users might think of these features as automatically available, app developers know that they must do work to integrate them. To make sure that an app provides the expected user experience for these features, developers follow iOS technology guidelines such as these:

• Support iCloud storage simply and transparently.
• Be prepared for multitasking-related interruptions and be ready to resume.
• Comply with the user’s Notification Center settings when handling local and push notifications.
• Supply descriptive information so that the app is accessible to VoiceOver users.
• Rely on the system-provided printing UI to enable a printing experience users appreciate.
• Ensure that sounds meet users’ expectations in all situations.

When an app correctly uses UI elements, such as buttons and tab bars, users are likely to notice only that the app behaves as they expect. But when an app uses UI elements incorrectly, users are often quick to voice their dissatisfaction. Great iOS apps take care to follow UI element usage guidelines. For example:

• Ensure that the back button in a navigation bar displays the title of the previous screen.
• Don’t remove a tab from a tab bar when its function is unavailable.
• Avoid providing a “dismiss popover” button.
• Always provide feedback when users select an item listed in a table view.
• On iPad, present a picker only within a popover.
• Use system-provided buttons and icons according to their documented meaning.
• When designing custom icons and images, use universal imagery that all users understand and avoid replicating Apple UI elements or products.

Again, the guidelines listed in this section represent a fraction of the guidelines contained in iOS Human Interface Guidelines. Reading that document in its entirety is an essential step in the app development process.

Take Advantage of Some Proven Design Strategies

The most successful iOS apps are often the result of thoughtful design iteration. When developers focus on the main task and continue to refine their list of features, they can create a superior user experience. The strategies summarized in this section can help you refine your idea, review design options, and converge on an app that people will appreciate.

Distill the feature list. As early as possible in the design process, define precisely what your app does and who your target audience is. Use this definition (called an application definition statement) to filter out unnecessary features and to guide the style of the app. Although it’s tempting to think that more features make a better app, the opposite is more often true. The best apps tend to focus on enabling a main task and providing only those features that users need to accomplish the task.

Design for the device. In addition to integrating the patterns of the iOS user interface and user experience, make sure that your app feels at home on the device. If you plan to develop a universal app (that is, an app that runs on both iPhone and iPad), this means that you must design a different UI for each device, even though much of the underlying code can be the same. Similarly, if you plan to start with web-based content, it’s essential that you redesign the content to look and feel like a native app.

Customize appropriately. Every app includes some UI customization, if only in its App Store icon. The iOS SDK gives you the ability to customize every aspect of the UI, so it’s up to you to decide how much customization is appropriate. The best apps balance customization with clarity of purpose and ease of use. Ideally, you want users to recognize the distinctiveness of your app, while at the same time appreciating the consistency that makes an app intuitive and easy to use.

Prototype and iterate. Soon after you’ve decided what features to include, begin creating testable prototypes. Early prototypes don’t need to display real UI or art and they don’t need to handle real content, but they do need to give testers an accurate idea of how the app works. During testing, pay particular attention to what testers try and fail to do, because those attempts can reveal places where the app appears to promise a behavior that it doesn’t deliver. Continue testing until you’re satisfied that users can easily grasp how the app works and operate all its features.

Don't waste my time

在辞职2个月后，居然越来越有活力。虽然理论上我应该萎靡不振才对。秘诀就是“Don't waste my time”。这句话是一个Phillip的同事送我的。在听到那句话的晚上，我居然气得头发晕，到了3，4点才睡。这真的是第一次这么生气。这么夸张？因为这句话是我正在加班加点的赶进度而故意Email给老板表明项目的Delay与之无关而说的。其实他不知道的是我早就已经向老板们表明，我愿意担负项目延误的责任了。而这位太监式的员工每天悠闲的准点上班下班，以前不时“好意”提醒我在这个公司，不需要太认真干活。就是这样一个人居然埋怨别人浪费了他的时间！！！我真的是彻底被打败了。而这倒是说出了一个我一直都没有搞明白的问题：Why others buy this？？ 原因就是Time。在公司我们总是有“敌人”，可能是老板，同事，下属。我们没有搞明白的是，真正的敌人其实是时间。时间不是我们的朋友。

在这过去的2个月中，当我必须为自己的生计发愁而必须真正学习牛X人的系统时才发现我原来错过了这么多好东西。真的是Waste My Time！！！而时间，就是在不停的揣测老板们的心情，不停的参加毫无意义的会议，不停地给完全不明白的外行重复解释系统的Design，不停地回答门外汉的Concern...中流失掉了。Fuck! 真的不愧蠢货的称号。那就是不知道自己是个蠢货。而牛人5年前就已经清楚的发布如何Hacking Iphone Apps 了http://blog.gauravgiri.com/2008/07/drm-fail/。 估计在牛人发布的时后，我正在跟傻逼解释Push/Poll的difference吧，晕倒， waste my time。。。

天道酬勤！过去的几年还是收获了很多管理上的经验。以后我应该是不可能在很上心的给外行解释什么是Push，什么是Poll了。也不会和一个太监式的人渣搅到一起，更不会花一下午的时间听太监的牢骚了。

而最宝贵的收获是我终于彻底明白了我自己的一个问题： Why others need buy your product？答案就是：时间。

我们每个人都在老去，死去。我们没有可能什么都自己做。所以任何可以节约我们时间的Product，我们都是愿意购买的。所以生意是什么？了解我们自己到底正在费力的干着什么，提供一个Product/Service来减少我们自己的麻烦，然后就可以卖了。而对于软件产品来说，现在有一个其他产品都没有的便利渠道，App Store！

千金难买明白！种种的郁闷也算值了！